A few days later, and we're back with another beta. But not with a lot more sleep, so I'm afraid we're going to have another relatively dry and factual post. With barely a parenthetical. No asides. Hardly any wind-up. Just the facts and nothing but. Pitched right down the middle. Put out there for you. Right in front of your face. No need to scroll. Hardly a word out of place. Nothing wasted. Not a single ounce of fat. Lean, tight prose.
Get to the Point
We've done a bunch more internal work on this beta, so (in general - hah! got a parenthetical in there) it should be cleaner and more functional for all. Specific changes include:
Erase, then copy now works with APFS When you're copying APFS to APFS, you can now use Erase rather than Smart Update, if you want to (or need to, because you're unregistered—but please register).
Improved bootability We've improved some edge case handling for some boot configurations.
Fixed non-expanding main window & copy now button In some configurations, the main window wouldn't expand, the copy now button wouldn't work, some elements of UI wouldn't reflect the current pop-up settings, etc. The root cause for all these problems has been fixed.
Drive UUIDs no longer shown instead of the name in some fields That's right, you didn't name your drive 3443-93YDAE-8834F-007EEDA, we goofed.
Source pop-up shows the size again And all is right with the world.
Better logging Mostly to help me when I'm helping you, which helps you, and me. Win-win.
Inconceivable
We had a few reports from people who were getting a very weird error during the bless process: they'd get a file not found error, and the backup would abort. With the help of some willing users (apologies and thanks to Jeff, Mark, Glen, Bryan, Michael and Paul), we thought the common element was FireWire, but then someone checked in with the same case with USB.
So we ruled out FireWire and pursued a bunch of different things, none of which worked.
Until we determined that the USB guy was having a different problem. Which meant all the others were FireWire. So, we asked for those who could to take the same drive, switch to USB, and Smart Update the result and... it worked.
Unbelievably, High Sierra won't bless APFS on FireWire, at least in its default configuration. We're trying to see if we can come up with a way around that, but until then, connect your FireWire drives via USB or Thunderbolt.
End of an era, folks.
UPDATE: We're having success with FireWire in-house, so it's definitely not all FireWire configurations. We're still trying to figure this out.
Just the Tips
Some things to keep in mind:
If you've just turned on encryption, your backups can't be performed until the encryption process is complete, since snapshots are disabled during the encryption process. So, sit back and let the encryption magic happen...then back up.
Please install SuperDuper! when you're an admin. Otherwise, the Quarantine attribute can get stuck on. Note that it's often easier to install by running SuperDuper! from its download image: it'll offer to install itself.
Don't convert a backup volume from HFS+ to APFS. Instead, erase it using the steps in the previous blog post.
Stop Yer Yappin'
See? Short post. Going to sleep now. Download away:
It's been a bit less than a week since High Sierra's release, and we've been busily updating SuperDuper! v3.0 Beta 1 to Beta 2 (which, per the usual custom, is available at the end of this post). Our thanks to all the users who took the time to download the first beta and provide feedback: it's been really helpful to have the additional coverage as we work to wrap up v3.0.
We've been really pleased with the way snapshots have been working with the new version. We haven't seen a single report of a problem due to highly active files: it's doing just what it's supposed to do, which is great. Doing it this way is going to really improve people's backup experience.
Sorry for the lack of jokes in this post. It's been a long week, and my punch-up crew is on braincation.
(Note that after a version of this post went up we found a better fix for one of the issues below; if you downloaded B2 at that time, please download and install it again.)
Format Change
A lot of people have been confused about how to format their backup drive as APFS, and are confused about how to get an HFS+ volume on the same drive as an APFS volume.
The new Disk Utility has some nice features, but they've buried a bunch of stuff in the UI. Here's how to do both.
Format the Whole Drive as APFS
In Disk Utility's "View" menu, "Show all devices".
Select the drive hardware, above the existing volume, in the sidebar.
Click the Erase button.
Choose the "GUID" partition scheme, and the plain APFS format.
Erase the drive.
Add an APFS partition to an existing drive
Select the external drive in the sidebar
Click the Partition tab
Click the "+" button below the partition diagram
Size the volume as needed
Choose the APFS format
Click Apply
Add a new APFS volume to an existing APFS container
Select an APFS volume in the sidebar that's in the container you want to add to
Select "Add APFS Volume" from the Edit menu
Select the options you want, including minimum and quota sizes if desired, and click Add
Now that you know how to do that, let's discuss SuperDuper-specific things.
We Go On Three?
Overall, the feedback has been quite positive, and I'm pretty happy with how v3.0 is working for most people, but (as expected - it's a beta) there are a few areas where we needed to either work around High Sierra weirdness, or fix our own bugs. Namely:
Copying from APFS to HFS+ fails
Three interesting things happened here. First, there's a special hidden folder on HFS+ called ".HFS Private Data^M" that we automatically ignore during a copy, since it's managed my HFS+ and is unique to the volume. And yes, it has a ^M character at the end of the name.
Weirdly, when High Sierra converts an HFS+ volume to APFS, it retains this particular folder, even though it's no longer something that's needed. And, on top of that, trying to match the name, with the ^M, fails with APFS "globbling"...and thus we copy the file.
Going APFS->APFS, this works fine, since the folder is just a folder. But going APFS->HFS+, we get an error, since it's not ignored (due to the APFS bug) and can't be written (because it shouldn't be copied).
We've worked around this in the new update.
There's no Preboot when going APFS->HFS+
A last minute change caused us to try to bless the Preboot volume when there wasn't one, since HFS+ volumes don't have preboot. Although we continue to recommend copying APFS to APFS, as discussed in my previous post, this is now fixed.
Errors when using Erase, then copy
Alas, erase copies of APFS volumes are failing when finalizing Preboot and Recovery. This happens because of an issue discussed in the previous post: the UUID of the volume changes during the erase, but we're copying the Preboot and Recovery based on the old UUID value, so they can't be found.
We didn't catch this because while we checked Erase with non-bootable volumes, to save time during our test pass, we stopped the bootable volume test and then checked Smart Update (since we were now going from the erased volume, and the previous erase tests passed). Dumb, and the test matrix has been updated to ensure it doesn't happen again. This will be fixed in the next beta: in the meantime, use Smart Update.
Crash with disk images
Another last minute regression (as we rolled back the image handling we didn't finish, mentioned in the last post). Fixed in the new beta.
Remember, though - if you're copying APFS volume to images, please open/mount the image first and copy to its mounted volume with Smart Update, rather than to the file.
Visible Snapshots
We consciously show the snapshot during the beta for debugging purposes, which confused some users. Given the good results we're seeing, they're now treated as release-ready, and are hidden.
Can't Create Snapshot
Two users had an interesting problem where the system returned that snapshots couldn't be created. It looks like this happens when the source drive is being encrypted but the process isn't complete. It's a very rare issue, given the number of testers, and the amount of coverage. We're continuing our investigation, and trying to reproduce in house.
If you're seeing this, and you haven't already contacted us, please get in touch!
Resource Busy error when updating Recovery
There are some interesting aspects to this particular problem. Based on the error, this should mean that the Recovery volume is mounted somewhere else, and we're trying to double-mount it. But we've checked for that, and it's not mounted anywhere else.
As we were further working through the case with some users who had it happen (we've never seen the problem in house), we also saw situations where some of the files in Recovery were busy during pruning, and thus couldn't be removed.
So, we came up with some clever workarounds which we've implemented in this version. They seem to work for our in-between-beta test users who hit the issue.
As much as we've love to say "problem solved", we're continuing to gather data, since it doesn't make a lot of sense, while also logging additional diagnostic information in case it does happen to the broader test group.
It's important to note that when this fails, your data has been backed up successfully, and you can restore: you just might not be able to boot from the backup. If you're stuck, as always, get in touch.
Various and sundry items
We fixed some problems caused by special characters in volume names, a few UI and log typos (no, we don't know what "Ingnorning" is either), updated the signature so it should open normally for all users, and the like.
Exit, Stage Right
So, that about does it for the Beta 2 release. Thanks again for helping out with the testing, and drop an email if you're having problems.
Told you it wouldn't be long until I posted again.
Spoiler Alert
We've finished up a bunch of internal testing over the past few weeks, and there's a beta of SuperDuper! for High Sierra and APFS linked at the bottom of this post. But it's so exciting, in a totally nerdy way, that it would be a mistake to not follow the whole story, with its twists and turns. So let's dive in.
Our Antagonist
As I mentioned in the last post, APFS is basically undocumented. That required us to make certain assumptions about its capabilities, and to draw inferences from what's on-disk in order to produce something that replicates the source.
APFS volumes are hosted in a container, which is basically a GUID partition on a drive. That container has a pool of storage which is shared between all the APFS volumes in the container. So, in a 1TB container, you could have a large number of volumes, each of which can draw from the 1TB of storage available in its container.
Only APFS volumes can be in the container: HFS+ volumes are not allowed.
In addition to the startup volume, and any volumes that you create yourself, a container that hosts bootable volumes also contains three hidden volumes with special "roles". One, called Preboot, contains all the information required to start the boot process. Another called Recovery, is the recovery volume. The third, VM, is where swap files are stored.
These volumes are minimally documented in the asr and diskutil man pages (the asr page actually discusses what makes up a valid system), so we're quite confident about what we've done in terms of the volumes and their roles. Unfortunately, their contents are not, and so here is where we've made assumptions. Also, since we can't create our own volume contents, we copy that data from the source's equivalent volumes.
Papers, Please
Both HFS+ and APFS volumes have their own unique ID (UUID). Each special volume mentioned above associates various information with the unique ID of the corresponding bootable, role-less APFS volume. Unfortunately, those unique IDs cannot be preserved across an erase: they're read-only. As such, there's significant housekeeping involved in maintaining these special volumes, and in positively identifying the volumes you've selected as the source and destination.
On top of that, the reboot and recovery data can't just be copied whole from the source, because multiple bootable APFS drives in a single container share the same special volumes... so, as you might expect, there's also new logic involved.
This all works its magic behind the scenes, and you shouldn't notice any changes...although you might notice some subtle improvements.
There and Back Again
As I mentioned in the previous post, we allow APFS to HFS+ copies (and back). However, copying from HFS+ requires the special volume contents to be stashed on the backup copy, since HFS+ volumes don't have all these special role volumes (and their partition schemes don't allow the same kind of 'guaranteed' partition creation). That's something we don't do in this first beta.
In addition, given the full extent of APFS's capabilities are not known, we're not fully confident in allowing this type of copying in our beta, and discourage the use of HFS+ to APFS copying. If you need to restore HFS+ to a bootable APFS volume, reinstall the OS to a clean volume and point to the backup when prompted during first boot.
And When You Smile for the Camera... (RIP Walter)
Ah, but these stories always have a twist! We think this is a good, Unbreakable twist, too.
One of the challenges of copying a live volume is that it's active and always changing. Even as we're stepping through the data, the user could delete files out from under us, change the data...almost anything can happen. In general, it doesn't. But it could. That's why the User's Guide suggests quitting applications before copying.
One awesome thing that APFS has is the ability to create "snapshots" that freeze the state of an APFS volume. Copies from the snapshot will exactly match the state of the source at the moment the snapshot is taken, so you don't have to worry about what applications are open, what might be written or deleted in the meantime, etc.
Snapshots have a lot of capabilities that were mentioned during the WWDC session, but most of those aren't yet available to developers. However, the ability to actually create and copy from them is both available and documented! So...
We're really happy to announce that SuperDuper! uses snapshots on APFS boot volumes when copying. Copies should be much more reliable and less error prone, and (as usual) the whole process happens completely transparently to you, the user.
To get this to work, there's literally nothing you have to do: no settings, no muss, no fuss. If your startup volume is an APFS volume, the details are all handled for you.
Continuity Errors
Apart from special-volume-stashing, we weren't able to finish implementing one thing in time for High Sierra's release: backing up to unmounted images. This also has to do with the special volumes mentioned earlier: when a volume isn't available, we don't know its format, nor do we know whether it requires volumes to be created.
We didn't want to mount the volume to find out, and there were so many cascading cases involved we decided to not release that part of what we've done. Instead, once you've created an APFS image backup, open it in Finder by double-clicking, then copy directly to the mounted image's volume. That'll create the special volumes so it can be properly restored.
You may also notice that we're copying a bit more than you'd expect: we see it too. We decided it was better to copy a little more than necessary than a little less, since it's just a matter of time taken and not overall fidelity. This will be improved in the final release.
The main application is now 64-bit, but the copy engine is still 32-bit. We didn't want to potentially destabilize the copy process by including our alpha-level 64-bit copy engine in a public beta, so we're 95% 64-bit, and 5% 32-bit.
Finally, there's a black window flash during animations that's related to the 64-bit changes. It's ugly. We know. But it's not a blocker, and there were more important things to get working (snapshots!) so, think of it as a brief moment to consider your place in the universe and...enjoy?
Dinner is Served
Since it's a beta, dinner is dog food. We've been eating it for a while now and it's delicious, but given FDA standards for dog-vs-people food, there may be a higher quantity of insect parts in your portion.
OK, that's kind of gross.
What I mean to say is that we've tested this internally, and we're pretty pleased with how it's working.
That said, we're talking about a brand new OS with a brand new file system and some brand new capabilities. This version is a beta, and while you can continue to use it as you normally would, you may encounter issues.
These things happen.
So, in conjunction, also use Time Machine as an extra backstop (always a good idea). I'm here for you at the support email; let me know if you have any problems.
We'll be working to move this from beta to final release as we finish up some of the remaining tasks, and the inspectors determine it's fit for human consumption. We might even get a chance to sleep a little. Probably not, though, so if any support replies are incoherent, please be understanding.
In the meantime, thanks for your patience and support: it's truly appreciated.
It's inevitable—another year, another macOS version (I am still not used to typing "macOS"), and another build up to a significant SuperDuper! release. Sorry I haven't been blogging the whole way through, but it's been busy, and I just haven't had the time.
As of this morning, we have a release date - September 25th. So, for those asking (and there are a lot of you), let's talk about High Sierra. Lucky 13! Woo!
APFS
First, we'll definitely be supporting APFS. That work has been in progress for some time, and continues as of this post. We already have copying to and from APFS volumes working "in the lab", as it were, and testing is ongoing.
The bad news is I'm not confident enough to say we're going to release our APFS support day-and-date.
I know this kind of hedging is disappointing. But it's important to note that Apple still hasn't released any documentation on the "proper" way to create a bootable APFS volume. An example of what they have in mind was released for the very first time when the High Sierra developer release came out a few months ago, but that's it. We basically have to make an educated guess about what they want.
We've designed and implemented that, and it's significantly different than HFS+'s boot setup, with various special partitions dedicated to specific purposes (even a separate VM volume!), and entire new volume management system, etc.
So, we have both copying and boot working, and we're testing it carefully to try to cover all of the various permutations that can occur. Because no guidance or documentation is available, and the process is all reverse-engineered from existing volumes, it's difficult to know what might happen in the future...and protecting against mistakes, and informing users what their potential actions mean, is really critical.
For example, what happens if you do an "Erase, then copy" from an HFS+ volume to an APFS volume? In our current version, we match the format of the source when we erase. But, HFS+ can't be in an APFS container. So, we'd have to convert the container to a regular GUID partition. And since there might be other APFS volumes in that container, you'd end up destroying them.
Not good. And that's just one case.
So, we're being cautious, knowing that users who have continually followed our advice (namely more backups are always better than fewer, and use more than one backup program, such as SuperDuper! and Time Machine rather than either on its own) will still be covered by Time Machine if they jump the High Sierra update as soon as it becomes available.
But what about HFS+?
Yes, making a backup from APFS to HFS+, and booting off that, can work as it always has (once our APFS support is released). And HFS+ to HFS+ works as expected as well, save for a few small issues in 2.9.1.
In particular, Apple has further tightened its System Integrity Protection process, and is completely denying access to some files on the startup volume, even when copying to a non-startup volume. We consider those errors ("Operation not permitted") to be fatal in 2.9.1, and so we stop. We have a simple, preference-based workaround for this that we've provided to people during the beta on request, and 2.9.2 has it built-in. (This is the kind of thing I'm talking about when I said "it's difficult to know what might happen"--we didn't know what could happen with SIP, and so we planned for possible changes.)
But since APFS is, again, basically undocumented... what could that backup be missing? How will changes like filename normalization affect future copies if a dot-update modifies behavior? How do you make a snapshot, since that is obviously intended to be a better source for a backup? And do androids actually dream of electric sheep?
These are the kinds of things that have been keeping us awake at night at Shirt Pocket HQ. I feel like I've aged as much as a two-term President.
Anyway, we're grinding through this all as quickly as we can. We don't want to finish until we're sure we have the shipping bits, because, unlike with earlier OS versions, we're dealing with an entire new file system. We'd rather be late than wrong.
We're probably not wrong, but we want to be sure.
Automatic conversion
Speaking of being sure, one of the scariest part of High Sierra is the fact that, if you have an SSD, you'll have no choice: you're going to be converted to APFS, whether you like it or not. Apple is obviously confident about this process, or they wouldn't do it, but even if the conversion is successful, it's important to note that while that means it worked from their point of view -- conversion accomplished! -- this changeover won't be without pain.
Your existing, valuable low-level tools, such as Disk Warrior, will instantly be rendered useless. In a pinch where Disk Utility can't fix it? Tough luck.
Want to host an HFS+ volume on your drive? You can't in an APFS container.
APFS doesn't seem to be faster than HFS+ (which is not to say it won't ever be, or that it won't be more stable...a low bar, I know). That cool demo where copying a whole directory is instantaneous? That's because the data isn't really being copied. It's being referenced, which is, of course, fast. Copying happens when you modify, so it's basically being done "lazily".
But think about it. That also means that if you made a copy of your data "just in case", and you developed a bad spot on the drive, all copies of the file are now damaged, because they share the same data. Disk damage can now cascade across far more files than before, because you can't tell what's shared. Even snapshots can't save you if they're referencing the same bucket of damaged bits.
So many other things.
There are going to be growing pains here, and a lot of it will be for long time developers who have been doing invaluable work for years, and unlike us, they have to start again from scratch. Need I say they deserve your support and encouragement?
64-bit
I'm also getting a lot of mail about our 64-bit plans, since there's been a pile of confusion coming out of the WWDC announcements, so let me try to set things straight.
High Sierra works fine with 32-bit apps. Come January, the App Store will stop accepting 32-bit store apps. And, at some point in the future, Apple will stop supporting 32-bit apps.
Now, for most apps, there's really no advantage to being 64-bit (save for some really minor system-wide memory benefits, which—frankly—were the other way around when 64-bit APIs were introduced, and no one complained about that). So for many developers this will be a lot of unnecessary work, with no end-user benefits.
All that said, while there's no rush to do so (they're not phasing out 32-bit for a while), our full High Sierra release will be 64-bit, so you can rest easy, knowing that we'll be ready for the actual phase-out well before it happens.
Should I Update to High Sierra?
I think I usually tell people to "take it slow" with an entirely new OS version. While there have been "huge win" updates in the past (10.6 is a great example), High Sierra isn't one of those.
Rather, while it may be a good (or even great) update over time, it also has the potential to destabilize things far more than anything that's come before. It's going to be hard to judge its impact until its wide release, and even then, it'll be a while.
If you're a normal user, I would strongly encourage you to not update to High Sierra right away. Let others take the risk. Wait until things calm down and the initial problems, which are inevitable, are fixed. Continue doing what you've been doing before High Sierra: you're not missing anything of significance.
Apple's going to be OK if their "adoption graph" doesn't go straight up in the air.
After all, why should you be a statistic when you have work to do?
But I Want to Update Right Away
Don't.
I Insist on Updating Immediately!
Sigh. OK. But don't say you haven't been warned: we don't all float down here.
Before you update, use SuperDuper to make a full backup or two. Check your backup by starting up from it. Then, unplug the backup drive from your Mac and put it somewhere safe.
As I've recommended elsewhere, you should be making multiple backups with other programs as well, including Time machine. So, after you update, continue to use Time Machine to back up. Given it's basically written by the same team as APFS, and Apple knows what is and isn't possible, even when they don't document it externally, this is your best bet to start with.
If you need to roll back to your previous OS, you cannot just use Smart Update. In fact, you also can't use Erase, then copy, because your internal drive has been converted to a storage scheme that won't work with anything earlier than High Sierra.
Surprise!
So, after starting up from your SuperDuper backup, use Disk Utility to erase the drive you want to restore to, by selecting the drive hardware above the volume, not the volume itself. That will roll you back to something that can properly host HFS+ and boot pre-High Sierra releases. You can then restore your SuperDuper! backup normally, using "Restore - all files" and "Erase, then copy" or "Smart Update".
But How Do I Back Up?
For the moment, as I said above, use Time Machine. Apple's teams know the details of APFS, and as such Time Machine isn't in the "limbo" we're in with regard to ensuring things are as they should be. Remember, disk space is cheap, and I know I say this too much, but it's true: no one was ever sad they had too many backups.
If you are running High Sierra with HFS+ (namely, if you have a spinning disk or Fusion drive), our just-released SuperDuper update will work fine. Use that in addition to Time Machine.
Our full, APFS-supporting version (with some other surprises) will be out as soon as we can get it finished. And we'll have a public beta when we're confident enough to put it in your hands.
This is Going to Cost Me a Ton, Isn't It.
Actually, no. Since 2004, we've never charged for SuperDuper updates, and our High Sierra update will be free of charge as well.
Clearly, we were dropped on the head as babies.
FIN
So there you go. I'll be back relatively soon with more.
I'm happy to say that we're releasing SuperDuper! v2.9 today, at long last. In fact, if you're reading this, it's out. Hooray!
This new release has a large number of enhancements and bug fixes that improve SuperDuper!'s responsiveness and performance. Many of these are longstanding items that required a significant amount of investigation, across many different systems, to diagnose and fix, so we're really grateful to the users who reported the problems and were willing to work with us to run down the underlying issues.
But rather than talk in detail about each (you can get the list of changes in Help > Release Notes), I want to discuss how SuperDuper! interacts with the bane of every copying application: Antivirus programs.
Anti-antivirus
Antivirus programs are getting more popular on the Mac, even while the threat environment doesn't get worse. And they're extending their reach, protecting against things like "phishing" emails. Plus, all Antivirus vendors include non-Mac threat signatures (mainly Windows applications) in with their Mac-specific applications, which tend to flag things that aren't really relevant.
The result of all this (beyond slowing copies down) is that Antivirus users are being informed of potential threats constantly, and before v2.9, each of these pseudo-threats would cause SuperDuper! to stop, because they're flagged as errors (specifically, "Permission Denied" errors) when we try to open the file for copying.
The vast majority of these files are in Junk mail folders. Due to the way Apple Mail (and similar applications) work, those files are cached locally. So, even though the user isn't really interacting with the mail, the files are on disk and copied, and that causes errors to happen much too often.
Given that error isn't Antivirus-specific, we've been reluctant to ignore them: our philosophy has always been to stop on errors (see the "I/O Error Handling" post) But, as more users install the apps, we've decided that the frequency of non-Antivirus related "Permission Denied" errors is so low (I haven't seen one in more than two years), the risk of ignoring these particular errors is minimal.
So, we've added a new preference -- which defaults to ON -- that ignores "Permission Denied" errors. We do log any occurrences, but we don't fail the copy.
This should significantly improve the lives of less technical users (or even technical ones who have Antivirus program use mandated by company policy).
macOS 10.12 - Sierra
To answer the inevitable question, we've been testing v2.9 on current Sierra builds, of course, and initial compatibility is looking good. Remember, though, Sierra is in beta. We do not have the final version to test against, anything can happen between now and Sierra's release, and as such we can't guarantee that we'll be fully compatible with the official, release version of macOS 10.12 until that happens.
Along similar lines, we're not (yet) APFS compatible. The documentation for APFS is still quite sparse, and given that it's in very early preview, we decided it was unwise to release an initially compatible version until it's a bit better documented and closer to done. (It's much easier for us to be pretty confident about the stability of the 8th beta of Sierra than these early releases of APFS.)
In the meantime, though, enjoy the new version, and don't hesitate to contact us if you have problems.
November has rolled around, and that means it's been a month or so since SuperDuper! was released for El Capitan. Rather than go through a litany of how awesome we are (I hope that's obvious), I thought I'd take a moment to discuss the things we've screwed up, bugs we've found, and other embarrassments. Prepare to judge us...harshly!
Update, schmupdate
First, as anticipated in previous blog posts, the broken automatic update (with an error at the end of the process, triggered by a last-minute change in System Integrity Protection, causing the update to fail) has been a never-ending series of hassles for our users (and a pretty unbelievable number of support requests).
Despite releasing the new version well before El Capitan came out, it's clear that people didn't update until they actually installed 10.11, and at that point the update failed.
Much of this is our fault. The update itself isn't displayed until the application launches, and a scheduled copy continues past the notice and completes the tasks at hand, then quits. That means that most users never even see SuperDuper! doing its thing, and thus never even know the update is available. So, by trying to be as "magic" and "unobtrusive" as possible, we end up hiding important information.
In most cases, this doesn't matter, since an update that is put out to work around a problem will be presented when a failure occurs. But in this case, it caused unnecessary stress, not to mention painful RSI flareups. (Not really, but man, I've been typing a lot on the new Magical Mystery Keyboard.)
Anyway, in the next major update we're definitely going to rework how this stuff happens. ("It's about time" shout the displeased and annoyed masses.)
Schedule screwup
This one's totally on me. I documented the problem with Apple's removal of /usr/bin/lockfile in this blog, but I forgot to add the fact that you have to delete and recreate your schedules to the update notice and release notes.
Not everyone on the 'net reads this blog. Who knew? (Me. Duh.)
Documenting Apple's Changes
El Capitan removed the ability for 3rd party applications to do certain things, and while we handle those cases well, I didn't actually document the changes in the release notes:
1. Repair Permissions is no longer an option (Apple removed it completely at the user/app level; it's been less than useful for many years now, so—as I indicated below—no big loss).
1. Non-Apple applications can no longer programmatically set the startup drive, so we can't offer that as an "On successful completion" option.
1. And, due to the above, the option to restart from the backup drive has also been disabled.
Save for some users' old habit of reflexively repairing permissions (we've always shipped with that option OFF), these are all used relatively rarely, so their loss is not keenly felt. Still, should have been in the release notes.
The Curious Case of the Tiny Pipe
Here's where I get a bit technical, so feel free to skip this section if you don't want the boring-ish details.
SuperDuper! is broken into a number of primary processes: a UI, an escalated privilege Agent, and the Copy Engine. (There are others, like the various parts of the scheduler, but those are the primary ones.)
Those processes communicate through Unix Pipes: basically, streams that run between two processes and allow commands to be issued and results returned. One process writes a task to the command pipe connected to the process it wants to control, and then reads the result from the result pipe.
This is how we've done things since the very first version of SuperDuper (and was, at the time, the Apple-sanctioned way of doing this kind of thing).
Works great, very Unix.
Weirdly, in El Capitan, we had a few users report that some commands—commands that are entirely static in our application, and issued to the shell—returned syntax errors. It didn't happen often, and we never saw it here, but when it did, the only thing that resolved the issue was restarting the Mac.
That's an annoying kind of problem, as you might guess.
We put together an instrumented series of builds for a user who was kind enough to run them (over and over and over), and determined that:
When in this state, the pipe was only passing through 512 bytes of the write. Powers of two: always suspicious.
No errors were returned from any of the write commands.
No exceptions were thrown.
The pipe wasn't buffered (or, we couldn't switch it to unbuffered if it was).
Attempts to try to flush the pipe didn't work.
Adding CRs (in case the pipe was somehow line-buffered) didn't do anything
You get the idea. (Again, thanks to the kind soul who ran this stuff over and over as we tried things out.)
So, we had a pipe with a weird size, that wasn't returning errors, but also wasn't passing through the data we were writing to it, once it was over 512 bytes.
Reviewing the POSIX documentation on pipes, they're usually pretty big (as in 64K), but the guaranteed size is only 512 bytes.
A number that might be spookily familiar. (Yes, I wrote this post on Halloween. Why do you ask?)
It seems that, on some systems, the size of a pipe write can shrink much smaller than in previous versions of the OS, perhaps because of resource constraints (although our test system that was always failing was a new Mac Pro with tons of resources)...as small, as far as we can tell, as the guaranteed minimum of 512 bytes. So, when our commands got larger than that, things started to fail on some systems, sometimes.
Still quite annoying. And even with that, we'd expect an error to be returned or thrown when writing to the pipe, but that didn't happen...and seems to be a new bug in El Capitan. Very hard to report, because we don't have a reproduction case: we've never seen it happen in house, on any system. But at least we had a way to work around the problem.
We now chunk our writes to whatever dynamic minimum we find we need. In testing, that allowed the data to pass through the pipe even when smaller than normal, and the syntax error (caused by the truncated data) went away.
Problem, hopefully, solved!
Not that bad
As new-release problems go, those are all annoying, but not too bad. Of course, we hate having any issues at all, but better minor than major! Please take that into consideration as you go into deliberations.
And the new release with better release notes and the pipe fix? A beta version is now available to you here!
Enjoy and, as always, thanks for being a SuperDuper! user.
It's not often that we get "official" final bits before Apple pushes the big red button that releases them to the public. I can't tell you how nice it is to be able to do the testing we need to do here before the email starts coming in, wondering when we're going to support the latest-and-greatest.
Well, sure, I guess I can tell you. It's nice. Really nice. Really, really nice. Thank you, Apple!
Wewease Bwian!
Because we've been able to test, I'm pleased to be able to provide those of you out there with the Release Candidate of SuperDuper! 2.8 as well. This has some minor changes as compared to the last update, based on the latest changes to El Capitan. It's nearly identical to what we're going to release next week to the public.
Not that you're not the public. Just that you're special, since you're reading this, and they're not.
Ch-ch-ch-changes
The main change is in our updater. Every task in SuperDuper is controlled by what we call a Transcript. The transcript coördinates the various parts of our code, Unix command-line tools, shell scripting, etc, to perform a task. One of those tasks is performing an auto-update.
Our automatic updater was one of the first out there, so it has a long history. And way back in the before time, OS X had a tendency to mess up permissions to certain folders when packages were installed. So, to help users out, our upgrade transcript fixed permissions for the Applications and Applications/Utilities folders.
The need for this diminished over the years, much like the need for repairing permissions, and we probably could have removed it. But it never hurt anything, and never caused a problem, so we didn't revisit the code.
Beta n
Of course, "never caused a problem" doesn't mean "never will cause a problem".
In a late Beta of El Capitan, "never caused" was automatically updated to "now causes", because changes to ownership and permissions for those folders (even if changed to the same values) became illegal...and that caused our transcript to fail.
We've fixed that in this update. But that fix isn't in the current version of SuperDuper, it's in version 2.8... which means users need to upgrade to 2.8 before they install El Capitan, or install manually.
We've adjusted the messaging in the update message to say this, so hopefully it won't be a big problem. But it won't be a problem for you because, again, special!
Wrapping up
That's about it! There are some other minor changes, but nothing to get worked up about. I think you'll find this version runs quite smoothly.
Public betas mean more storytelling on the blog. I feel a bit like we're developing in public—something I don't really like doing—but I hope it's interesting to those of you reading the blog and testing both El Capitan and these public SuperDuper releases.
One at a time, please
Ever since Ye Olden Days of OS X, there's been a handy Unix command called "lockfile"—in /usr/bin—that creates semaphores. It's quite handy in scripts of various sorts to ensure that "critical" sections are only executed by one script at a time, if more than one instance of that script might be running simultaneously. Scripts could be anything with shell access: shell scripts, perl scripts, AppleScripts...whatever.
We've used this for a long time in SuperDuper's schedule driver (which has always been written in AppleScript as an example of how to use SuperDuper!'s handy scripting interface); since you could have any number of schedules running at any given time, there are some operations that need to be atomic, and lockfile facilitated that. Basically, all the instances of the schedule scripts coöperate (you're welcome, New Yorker style guide, I hope your writers no longer feel so alone) using a semaphore that ensures only one runs at a time.
And all was great in the world.
Meep meep
Little did we know that there was a shoe—or an ACME-branded anvil—preparing to drop (although perhaps the fact that it was in /usr/bin was a hint): in El Capitan B4, Apple decided to stop shipping Procmail, and with it, lockfile. It wasn't deprecated and then removed... it was unceremoniously sent to the bit bucket. So, as of B4, scheduling in El Capitan broke.
There isn't any equivalent command in OS X that we could find, but there were a number of methods we could have used to fix this:
Use a complicated shell script, along with the seemingly-atomic mkdir command, to create and manage our own semaphore, using similar semantics
Add a command to our AppleScript dictionary to implement a semaphore
Add some commands to another utility program we have to implement semaphores
Create our own separate lockfile-equivalent utility
Engage in a futile effort to get lockfile put back into OS X (not going to happen, of course, since the change was obviously intentional; indeed, our bug was closed "as intended")
Crawl into a dark room and cry softly, hoping the problem resolves itself
There's no crying in OS X, mostly
In the end, we did none of those things. Instead, since Procmail is Open Source, we changed our build process to build lockfile as well, and included that command, unmodified, in our bundle.
Et Voilà!
This also meant we had to change the scripts that were looking for lockfile to find our application bundle and call the new, "local" version of lockfile instead. And that means, unfortunately, that users have to delete and recreate their schedules.
I hate it when that's necessary. Really. Not just because it means lots of additional tech support (people don't usually read the release notes). I hate it because it's a pain, even though it's not hard, and it means an update isn't as transparent as I want it to be. New versions of SuperDuper! should drop in, require no work, and present no surprises to the user, other than pleasant ones. Unfortunately, sometimes reality gets in the way.
The great reward
So without further ado, download Beta 3 - and don't forget to delete and recreate your schedules!
If it's not one thing, it's nothing, at least this time.
We'd developed the current Beta of SuperDuper using the most current Developer Beta versions, based on the feedback of various testers (and, of course, our own testing).
As I mentioned in my last blog post, we'd researched the problem we had copying El Capitan, and we came up with a way of getting the drive copied, although System Protection was disabled on the copy until the OS was reinstalled.
We thought it was critical to release that as soon as the Public Beta was announced, to ensure that the larger pool of public testers had access to it - we didn't want that audience to go without the ability to back up. And it's been working great.
Turns out...
Apple fixed the problem with copying the "com.apple.rootless" attribute in the Public Beta! So, with the release of our Beta 2 (download below), we've included the ability to copy with that EA preserved, and thus system protection is maintained on the copy as well. Plus, there's no need to erase when restoring.
This is all great news for users: basically, copying will work as it always has.
Every new OS X release has its own special challenges, and OS X 10.11 - which I still have trouble referring to as "El Capitan" - is no different. And in our testing (which we commenced immediately upon availability of the developer preview), we found that we couldn't make a copy of an El Capitan disk due to the new system protection or rootless feature.
Rootless mode is a good thing, for the most part. It makes OS X more secure by protecting various system folders, ensuring that even applications that obtain escalated privileges through trickery (or hacking) can't mess with these critical locations. (It also means that jailbreaking iOS 9 is likely to be much more difficult, for those who care about that kind of thing.)
In our investigation, we've found that a new Extended Attribute -- com.apple.rootless -- is used to mark files and folders with this new protection. No process other than certain Apple-signed-and-authored ones can remove or write this attribute, and files and folders marked with this attribute cannot be changed. However, those files and folders can be read and copied.
This means that, for those using El Capitan, we can't hint obliquely that we're compatible, as we have in the past, where our current version worked even though we couldn't declare that compatibility until the final build. This time, the current version of SuperDuper is dead in the water on El Capitan. It just won't work.
But don't dismay: we've worked to change that. I'm happy to say, to those of you who are on the Beta (and those who are going to join the public beta today), we've developed and tested a Beta version of SuperDuper that makes bootable copies of El Capitan. There's a link to download it at the end of this post.
But please don't skip down there. Keep reading.
There are a few minor caveats, and some things to keep in mind.
First, and most importantly, OS X 10.11 is in Beta, and so is this build of SuperDuper. While we've been super careful about changing as little as possible, El Capitan is a big update, and there may be things that don't work. There may be things that don't work as well as you'd like. If that's the case, report the problem to the appropriate party. We're happy to get the feedback, and I'm sure Apple is as well.
Operationally, there are some known, minor issues. The most inconsequential one is that Repair Permissions is no longer available under El Capitan, so we disable it in Options. I can't say it will be missed.
Since we can't write the com.apple.rootless EA, SuperDuper removes it during the copy. That means the backup -- while fully functional and bootable -- is not an "exact copy" of the source. Specifically, SuperDuper! must disable the system protection feature on the backup, and cannot recreate it when you restore.
That's a relatively minor difference, but it's an important one. After restore, your system becomes vulnerable to the kinds of attacks that Apple is specifically protecting against.
It's easy to regain full system protection features: you simply need to reinstall the OS from the App Store. You can do this at your leisure, but doing it as soon as possible means you're less vulnerable (even though that vulnerability is quite small). It's a painless process, and it writes the fresh OS under your existing applications and data. As an added benefit, it will speed up your boot process, since it'll recreate certain caches that non-special-Apple-programs can no longer update.
Also, for this version, if you want to restore over an existing El Capitan install, and there are changes in protected system folders, you cannot use Smart Update (because we can't overwrite those protected files, or write to those protected folders). We're hoping to remove this restriction in the next release.
That's it for now! Thanks, as always, for using and recommending SuperDuper: we appreciate it, and couldn't do it without your support.
