November has rolled around, and that means it's been a month or so since SuperDuper! was released for El Capitan. Rather than go through a litany of how awesome we are (I hope that's obvious), I thought I'd take a moment to discuss the things we've screwed up, bugs we've found, and other embarrassments. Prepare to judge us...harshly!
Update, schmupdate
First, as anticipated in previous blog posts, the broken automatic update (with an error at the end of the process, triggered by a last-minute change in System Integrity Protection, causing the update to fail) has been a never-ending series of hassles for our users (and a pretty unbelievable number of support requests).
Despite releasing the new version well before El Capitan came out, it's clear that people didn't update until they actually installed 10.11, and at that point the update failed.
Much of this is our fault. The update itself isn't displayed until the application launches, and a scheduled copy continues past the notice and completes the tasks at hand, then quits. That means that most users never even see SuperDuper! doing its thing, and thus never even know the update is available. So, by trying to be as "magic" and "unobtrusive" as possible, we end up hiding important information.
In most cases, this doesn't matter, since an update that is put out to work around a problem will be presented when a failure occurs. But in this case, it caused unnecessary stress, not to mention painful RSI flareups. (Not really, but man, I've been typing a lot on the new Magical Mystery Keyboard.)
Anyway, in the next major update we're definitely going to rework how this stuff happens. ("It's about time" shout the displeased and annoyed masses.)
Schedule screwup
This one's totally on me. I documented the problem with Apple's removal of /usr/bin/lockfile in this blog, but I forgot to add the fact that you have to delete and recreate your schedules to the update notice and release notes.
Not everyone on the 'net reads this blog. Who knew? (Me. Duh.)
Documenting Apple's Changes
El Capitan removed the ability for 3rd party applications to do certain things, and while we handle those cases well, I didn't actually document the changes in the release notes: 1. Repair Permissions is no longer an option (Apple removed it completely at the user/app level; it's been less than useful for many years now, so—as I indicated below—no big loss). 1. Non-Apple applications can no longer programmatically set the startup drive, so we can't offer that as an "On successful completion" option. 1. And, due to the above, the option to restart from the backup drive has also been disabled.
Save for some users' old habit of reflexively repairing permissions (we've always shipped with that option OFF), these are all used relatively rarely, so their loss is not keenly felt. Still, should have been in the release notes.
The Curious Case of the Tiny Pipe
Here's where I get a bit technical, so feel free to skip this section if you don't want the boring-ish details.
SuperDuper! is broken into a number of primary processes: a UI, an escalated privilege Agent, and the Copy Engine. (There are others, like the various parts of the scheduler, but those are the primary ones.)
Those processes communicate through Unix Pipes: basically, streams that run between two processes and allow commands to be issued and results returned. One process writes a task to the command pipe connected to the process it wants to control, and then reads the result from the result pipe.
This is how we've done things since the very first version of SuperDuper (and was, at the time, the Apple-sanctioned way of doing this kind of thing).
Works great, very Unix.
Weirdly, in El Capitan, we had a few users report that some commands—commands that are entirely static in our application, and issued to the shell—returned syntax errors. It didn't happen often, and we never saw it here, but when it did, the only thing that resolved the issue was restarting the Mac.
That's an annoying kind of problem, as you might guess.
We put together an instrumented series of builds for a user who was kind enough to run them (over and over and over), and determined that:
- When in this state, the pipe was only passing through 512 bytes of the write. Powers of two: always suspicious.
- No errors were returned from any of the write commands.
- No exceptions were thrown.
- The pipe wasn't buffered (or, we couldn't switch it to unbuffered if it was).
- Attempts to try to flush the pipe didn't work.
- Adding CRs (in case the pipe was somehow line-buffered) didn't do anything
- You get the idea. (Again, thanks to the kind soul who ran this stuff over and over as we tried things out.)
So, we had a pipe with a weird size, that wasn't returning errors, but also wasn't passing through the data we were writing to it, once it was over 512 bytes.
Reviewing the POSIX documentation on pipes, they're usually pretty big (as in 64K), but the guaranteed size is only 512 bytes.
A number that might be spookily familiar. (Yes, I wrote this post on Halloween. Why do you ask?)
It seems that, on some systems, the size of a pipe write can shrink much smaller than in previous versions of the OS, perhaps because of resource constraints (although our test system that was always failing was a new Mac Pro with tons of resources)...as small, as far as we can tell, as the guaranteed minimum of 512 bytes. So, when our commands got larger than that, things started to fail on some systems, sometimes.
Still quite annoying. And even with that, we'd expect an error to be returned or thrown when writing to the pipe, but that didn't happen...and seems to be a new bug in El Capitan. Very hard to report, because we don't have a reproduction case: we've never seen it happen in house, on any system. But at least we had a way to work around the problem.
We now chunk our writes to whatever dynamic minimum we find we need. In testing, that allowed the data to pass through the pipe even when smaller than normal, and the syntax error (caused by the truncated data) went away.
Problem, hopefully, solved!
Not that bad
As new-release problems go, those are all annoying, but not too bad. Of course, we hate having any issues at all, but better minor than major! Please take that into consideration as you go into deliberations.
And the new release with better release notes and the pipe fix? A beta version is now available to you here!
Enjoy and, as always, thanks for being a SuperDuper! user.
