Smarty Pants Monday, September 24, 2018
Executive Summary
SuperDuper 3.2 is now available. It includes
In the Less Smart Days of Old(e)
Since the SuperDuper!'s first release, we've had Smart Update, which speeds up copying by quickly evaluating a drive on the fly, copying and deleting where appropriate. It does this in one pass for speed and efficiency. Works great.
However, there's a small downside to this approach: if your disk is relatively full, and a change is made that could temporarily fill the disk during processing, even though the final result would fit, we're trigger a disk full error, and stop.
Recovery typically involved doing an Erase, then copy backup, which took time and was riskier than we'd like.
Safety First (and second)
There are some subtleties in the way Smart Update is done that can aggravate this situation -- but for a good cause.
While we don't "leave all the deletions to the end", as some have suggested (usually via a peeved support email), we consciously delete files as late as is practical: what we call "post-traversal". So, in a depth-first copy, we clean up as we "pop" back up the directory tree.
In human (as opposed to developer) terms, that means when we're about to leave a folder, we tidy it up, removing anything that shouldn't be there.
Why do we do it this way?
Well, when users make mistakes, we want to give them the best chance of recovery with a data salvaging tool. By copying before deleting at a given level, we don't overwrite them with new data as quickly. So, in an emergency, it's much easier for a data salvaging tool to get the files back.
The downside, though, is a potential for disk full errors when there's not much free space on a drive.
Smart Delete
Enter Smart Delete!
This is something we've been thinking about and working on for a while. The problem has always been balancing safety with convenience. But we've finally come up with a idea (and implementation) that works really well.
Basically, if we hit a disk full error, we "peek" ahead and clean things up before Smart Update gets there, just enough so it can do what it needs to do. Once we have the space, Smart Delete stops and allows the regular Smart Update to do its thing.
Smart Update and Smart Delete work hand-in-hand to minimize disk full errors while maximizing speed and safety, with no significant speed penalty.
Everyone Wins!
So there you go: another completely "invisible" feature that improves SuperDuper! in significant ways that you don't have to think about...or even notice. You'll just see (or, rather, not see) fewer failures in more "extreme" copies.
This is especially useful for Photographers and others who typically deal with large data files, and who rename or move huge folders of content. Whereas before those might fill a drive, now the copy will succeed.
Mojave Managed
We're also supporting Mojave in 3.2 with one small caveat: for the moment, we've opted out of Dark Mode. We just didn't have enough time to finish our Dark Mode implementation, didn't like what we had, and rather than delay things, decided to keep it in the lab for more testing and refinement. It'll be in a future update.
More Surprises in Store
We've got more things planned for the future, of course, so thanks for using SuperDuper! -- we really appreciate each and every one of you.
Enjoy the new version, and let us know if you have any questions!
Download SuperDuper! 3.2
3.2 B3: The Revenge! Wednesday, September 12, 2018
(OK, yeah, I should have used "The Revenge" for B4. Stop being such a stickler.)
Announcing SuperDuper 3.2 B3: a cavalcade of unnoticeable changes!
The march towards Mojave continues, and with the SAE (September Apple Event) happening today, I figured we'd release a beta with a bunch of polish that you may or may not notice.
But First...Something Technical!
As I've mentioned in previous posts, we've rewritten our scheduling, moving away from AppleScript to Swift, to avoid the various security prompts that were added to Mojave when doing pretty basic things.
Initially, I followed the basic structure of what I'd done before, effectively implementing a fully functional "proof of concept" to make sure it was going to do what it needed to do, without any downside.
In this Beta, I've moved past the original logic, and have taken advantage of capabilities that weren't possible, or weren't efficient, in AppleScript.
For example: the previously mentioned com.shirtpocket.lastVolumeList.plist was a file that kept track of the list of volumes mounted on the system, generated by sdbackuponmount at login. When a new mount occurred, or when the /Volumes folder changed, launchd would run sdbackuponmount again. It'd get a list of current volumes, compare that to the list of previous volumes, run the appropriate schedules for any new volumes, update com.shirtpocket.lastVolumeList.plist and quit.
This all made sense in AppleScript: the only way to find out about new volumes was to poll, and polling is terrible, so we used launchd to do it intelligently, and kept state in a file. I kept the approach in the rewritten version at first.
But...Why?
When I reworked things to properly handle ThrottleInterval, I initially took this same approach and kept checking for new volumes for 10 seconds, with a sleep in between. I wrote up the blog post to document ThrottleInterval for other developers, and posted it.
That was OK, and worked fine, but also bugged me. Polling is bad. Even slow polling is bad.
So, I spent a while reworking things to block, and use semaphores, and mount notifications to release the semaphore which checked the disk list, adding more stuff to deal with the complex control flow...
...and then, looking at what I had done, I realized I was being a complete and utter fool.
Not by trying to avoid polling. But by not doing this the "right way". The solution was staring me right in the face.
Block-head
Thing is, volume notifications are built into Workspace, and always have been. Those couldn't be used in AppleScript, but they're right there for use in Objective-C or Swift.
So all I had to do was subscribe to those notifications, block waiting for them to happen, and when one came in, react to it. No need to quit, since it's no longer polling at all. And no state file, because it's no longer needed: the notification itself says what volume was mounted.
It's been said many times: if you're writing a lot of code to accomplish something simple, you're not using the Frameworks properly.
Indeed.
There really is nothing much more satisfying than taking code that's become overly complicated and deleting most of it. The new approach is simpler, cleaner, faster and more reliable. All good things.
Download
That change is in there, along with a bunch more. You probably won't notice any big differences, but they're there and they make things better.
Download SuperDuper! 3.2 B3
Warning: this is a technical post, put here in the hopes that it'll help someone else someday.
We've had a problem over the years that our Backup on Connect LaunchAgent produces a ton of logging after a drive is attached and a copy is running. The logging looks something like:
9/2/18 8:00:11.182 AM com.apple.xpc.launchd[1]: (sdbackuponmount) Service only ran for 0 seconds. Pushing respawn out by 60 seconds.
Back when we originally noticed the problem, over 5 years ago, we "fixed" it by adjusting ThrottleInterval to 0 (found experimentally at the time). It had no negative effects, but the problem came back later and I never could understand why...certainly, it didn't make sense based on the man page, which says:
ThrottleInterval <integer>
This key lets one override the default throttling policy imposed on jobs by launchd. The value is in seconds, and by default, jobs will not be spawned more than once every 10 seconds. The principle behind this is that jobs should linger around just in case they are needed again in the near future. This not only reduces the latency of responses, but it encourages developers to amortize the cost of program invocation.
So. That implies that the jobs won't be spawned more often than every n seconds. OK, not a problem! Our agent processes the mounts changes quickly, launches the backups if needed and quits. That seemed sensible--get in, do your thing quickly, and get out. We didn't respawn the jobs, and processed all of the potential intervening mounts and unmounts that might happen in a 10-second "throttled" respawn.
It should have been fine... but wasn't.
The only thing I could come up with was that there must be a weird bug in WatchPaths where under some conditions, it would trigger on writes to child folders, even though it was documented not to. I couldn't figure out how to get around it, so we just put up with the logging.
But that wasn't the problem. The problem is what the man page isn't saying, but is implied in the last part: "jobs should linger around just in case they are needed again" is the key.
Basically, the job must run for at least as long as the ThrottleInterval is set to (default = 10 seconds). If it doesn't run for that long, it respawns the job, adjusted by a certain amount of time, even when the condition isn't triggered again.
So, in our case, we'd do our thing quickly and quit. But we didn't run for the minimum amount of time, and that caused the logging. launchd would then respawn us. We wouldn't have anything to do, so we'd quit quickly again, repeating the cycle.
Setting ThrottleInterval to 0 worked, when that was allowed, because we'd run for more than 0 seconds, so we wouldn't respawn. But when they started disallowing it ("you're not that important")...boom.
Once I figured out what the deal was, it was an easy enough fix. The new agent runs for the full, default, 10-second ThrottleInterval. Rather than quitting immediately after processing the mounts, it sleeps for a second and processes them again. It continues doing this until it's been running for 10 seconds, then quits.
With that change, the logging has stopped, and a long mystery has been solved.
This'll be in the next beta. Yay!
Technical Update! Thursday, September 06, 2018
SuperDuper! 3.2 B1 was well received. We literally had no bugs reported against it, which was pretty gratifying.
So, let's repeat that with SuperDuper! 3.2 B2! (There's a download link at the bottom of this post.)
Remember - SuperDuper! 3.2 runs with macOS 10.10 and later, and has improvements for every user, not just those using Mojave.
Here are some technical things that you might not immediately notice:
If you're running SuperDuper! under Mojave, you need to add it to Full Disk Access. SuperDuper! will prompt you and refuse to run until this permission has been granted.
Due to the nature of Full Disk Access, it has to be enabled before SuperDuper is launched--that's why we don't wait for you to add it and automatically proceed.
As I explained in the last post, we've completely rewritten our scheduling so it's no longer in AppleScript. We've split that into a number of parts, one of which can be used by you from AppleScript, Automator, shell script--whatever--to automatically perform a copy using saved SuperDuper settings.
In case you didn't realize it: copy settings, which include the source and destination drives, the copy script and all the options, plus the log from when it was run, can be saved using the File menu, and you can put them anywhere you'd like.
The command line tool that runs settings is called sdautomatedcopycontroller (so catchy!) and is in our bundle. For convenience, there's a symlink to it available in ~/Library/Application Support/SuperDuper!/Scheduled Copies, and we automatically update that symlink if you move SuperDuper.
The command takes one or more settings files as parameters (either as Unix paths or file:// URLs), and handles all the details needed to run SuperDuper! automatically. If there's a copy in progress, it waits until SuperDuper! is available. Any number of these can be active, so you could throw 20 of them in the background, supply 20 files on the command line: it's up to you. sdautomatedcopycontroller manages the details of interacting with SuperDuper for you.
- We've also created a small Finder extension that lets you select one or more settings files and run them--select "Run SuperDuper! settings" in the Services menu. The location and name of this particular command may change in future betas. (FYI, it's a very simple Automator action and uses the aforementioned
sdautomatedcopycontroller.)
We now automatically mount the source and destination volumes during automated copies. Previously, we only mounted the destination. The details are managed by sdautomatedcopycontroller, so the behavior will work for your own runs as well.
Any volumes that were automatically mounted are automatically scheduled for unmount at the end of a successful copy. The unmounts are performed when SuperDuper quits (unless the unmount is vetoed by other applications such as Spotlight or Antivirus).
- There is no #5.
sdautomatedcopycontroller also automatically unlocks source or destination volumes if you have the volume password in the keychain.
If you have a locked APFS volume and you've scheduled it (or have otherwise set up an automated copy), you'll get two security prompts the first time through. The first authorizes sdautomatedcopycontroller to access your keychain. The second allows it to access the password for the volume.
To allow things to run automatically, click "Always allow" for both prompts. As you'd expect, once you've authorized for the keychain, other locked volumes will only prompt to access the volume password.
We've added Notification Center support for scheduled copies. If Growl is not present and running, we fall back to Notification Center. Our existing, long-term Growl support remains intact.
If you have need of more complicated notifications, we still suggest using Growl, since, in addition to supporting "forwarding" to the notification center, it can also be configured to send email and other handy things.
Plus, supporting other developers is cool. Growl is in the App Store and still works great. We support 3rd party developers and think you should kick them some dough, too! All of us work hard to make your life better.
Minor issue, but macOS used to clean up "local temporary files" (which were deleted on logout) by moving the file to the Trash. We used a local temporary file for Backup on Connect, and would get occasional questions from users asking why they would find a file we were using for that feature in the trash.
Well, no more. The file has been sent to the land of wind and ghosts.
That'll do for now: enjoy!
Download SuperDuper! 3.2 B2
Page 1 of 1 pages
