Shirt Pocket Watch

In the end it was respiratory, as we knew it would be.

As I wrote yesterday, Ketzl had a bad night, and we couldn’t quite figure out what was going on.

But we should have known. After each loss of function, Ketzl became anxious, at least until she got used to the fact that this-or-that wasn’t going to work any more, and adjusted.

But it’s hard to adjust to a diaphragm that’s just isn’t working well enough to draw sufficient air. The heart tries to compensate by increasing its own rate, rushing the blood by the lungs to try to keep levels as high as it can, but it can only keep that up so long. And it can’t be comfortable, a 190bpm rate.

A bad night.

Followed by a pretty low day. She was tired—we all were—and lay next to me in my home office on egg crate foam I covered with a sheet, propped by pillows, trying to rest. She wouldn’t eat—never a good sign—but would take water. Occasionally, she would sleep, head sideways on a pillow, mouth open, panting. I did what I could to make her comfortable.

Later in the afternoon, to try to perk her up a bit, I took her for a drive. Windows down in the car, a cool breeze ruffling her fur. She lay in the back on the seat, raising her head occasionally to sniff, to try to look, and put it down on her paws. And panted.

Back home, I carried her downstairs and placed her on the floor, near the cool breeze from a fan. On her left side, she gasped for breath, clearly struggling. Thinking I’d put her down wrong, I propped her a bit more sternal and put her head on my lap. I checked her airway, which was clear, and got her water, which she drank. Her head rested on my thigh, on the floor. I went upstairs, put some ice in a towel and wiped her ears, pads, muzzle.

Around 7:30pm, Zabeth called on her way back from rounds. I told her she should hurry home, since something was wrong. Or tried to. Ketzl rested on my lap.

When she got here, we talked, cried, knew what had to be done. Originally, our cut-off point was going to be when she lost the use of her front legs, but when that happened a few months ago, Ketzl stayed bright, alert and happy. So, instead, we waited—waited for a sign from Ketzl.

Given, given.

I carried Ketzl back to the car and we drove to VESCONE. I had wanted to do this with our long-time vet, but it wasn’t to be: I couldn’t let Ketzl suffer for another day. VESCONE was nearby, and they had a nice room with carpet, a couch, fish. An attempt to provide a comforting space. Better than a stainless steel table.

Ketzl was placed on a gurney, a nylon strap holding her in place, a formality, given the fact that she really couldn’t move. She was comfortable, and had been given oxygen to help her breathe more normally. More relaxed, we gave her some turkey, some cheese. She ate for us, her last meal.

And we hugged her, and cried, and the Pink Hammer came down, and she was gone. Her eyes remained open, sunken, lifeless. I tried to close them—the freezer is a harsh place—but couldn’t. Icy eyes. I’m sorry, Ketzl.

We cut three tufts of fur and bundled them like sheaves of wheat: glossy black, chestnut brown and white. Bright white, the color of new snow, the snow she loved so. I wish I could have given her just one more glorious day.

We left the room, with her on the gurney. I closed the door and left the fish to watch over her, left the waiting room, desperately trying not to make eye contact with the people there.

At home, Z and I cried over a glass of wine, raised a toast to Ketzl. She was a great dog, and shared her life with us for over nine years, good times and bad. We loved her like a child, knew this was part of the bargain.

The light that burns twice as bright lasts half as long.

And she burned so very, very brightly.

Thanks, Ketzl, for the light you brought into our lives.

A rough night.

Ketzl, after a relatively calm day, decided that last night was a good night to demand water every two minutes and cry until she got it. This is pretty irrational behavior, and we’re concerned that DM has reached her brain stem and might be starting to affect her higher functions—something that will happen eventually.

No other weird signs, though, but it’s hard to tell: I’m feeling pretty irrational myself on a total lack of sleep. And, the day is pretty fully booked with stuff, so no real time to try to get a brief nap in.

If you get a reply to email today that looks like gibberish, you now know why.

With Ketzl on the decline, every day bring a new challenge, a new obstacle to overcome, a new way to show her how much we love her by making her life the best we can, given the circumstances.

As DM has moved forward, it’s affected all the muscles along the way. Recently, it’s affected her facial muscles. When she lies down, her upper lip tends to work its way into her mouth, and when she sleeps, she bites it. She’s stoic, but I’m sure it’s painful.

We’ve tried to come up with a way to deal with this, and Zabeth hit upon an approach that works, for the most part.

Z took two wide picture hanging hooks, re-bent them and smoothed the edges. Combined with a shoelace, we can hook her upper list on both sides and tie the shoelace over the top of her muzzle. It looks a little silly—kind of like braces—but it holds up her lip just enough… and it’s healing.

Ah, necessity…

Ketzl loves cows.

I’m sure this is related to the fact that Berners were farm dogs, and cattle herders (or descended from them), but whatever—cows make her excited and happy. A state I try to induce as often as I can.

Saturday, Zabeth was taking the last real day of her brief break to canoe in New Hampshire as I watched Ketzl. (She’s been helping for the last two weeks, which has been a real relief—vet school doesn’t leave her much time for anything other than rounds and sleep these days, and it was wonderful to have her here for a week or two.)

So come afternoon—it was Cow Time.

Stonehedge Farm is a “gentleman’s farm” just up the road—“gentleman’s farm” in the “rich people who own lots of land and have hired someone to raise cows for them” sense. It’s a nice bit of property on Waltham Road (and getting bigger all the time), and their cows are friendly and hang out near the road.

As I’ve mentioned before, K’s now effectively a quadriplegic: she can’t move three limbs and has minimal motion in the fourth. Her neck still works, though, so—placed properly—she can hold her head up and look around.

I took her out of the back seat of the car—no mean feat, 75 pounds of dead weight (down from 96)—and carried her over to the 10-odd cows hanging out by a closed gate. Propped her nicely and she totally engaged with the big beasts, chewed grass with them, watched intently—intensely focused. They knew she wasn’t a threat but remained a bit wary: wolf-things can be tricky, you know.

Ketzl looked at me, and the cows, and the grass, and me again. Chewed some more long stalks. Cows are, clearly, a good thing.

After a while the flies started to bother her. I hauled her back into my arms (this is going to kill my back) and walked up the road back toward the car.

On the way, another driver stopped and rolled down the window.

“Oh my god! What happened? Is that dog OK?”

I guess I’m used to this whole thing, and don’t think about how it looks to others: it took me a minute to figure out he thought I hit her with my car.

“Oh, it’s just fine. She’s paralyzed.”

He drove off, a bit confused.

Back in the car, driving home the mile or two, radiator blows in a weird, mild, was-able-to-make-it-home kind of way. (Must you fall apart too, car?)

Ketzl, inside and lying on her bed as sternal as I could manage to put her, sleeps.

And dreams.

Of cows.

A quick follow-up to my review of the Roomba and US Trilobite.

As I mentioned in the review, Europe has a newer version of the Trilobite—v2.0. The description of the 2.0 version from Electrolux indicated 200 changes from v1.0 (which was Europe-only as well), but despite communications with a number of Electrolux representatives, I wasn’t able to get a list of the differences between the US EL520 model and v2.0.

Based on two particular items—scheduling and smarter navigation—and after ensuring the v2.0 model was compatible with the US chargers and accessories—I sourced one from the UK. (Interestingly, the list price of the Trilobite in Europe is much less than in the US.)

The 2.0 Trilobite’s software is clearly a later revision, and has a number of behavioral differences, as well as scheduling and a graphical “time to charge” indicator. Although advertised as adjusting its cleaning time based on obstacles found in the room, it does not seem to act any differently (in this area) than the EL520. Most of the mechanical design seems to be identical to the EL520: I believe that the vast majority of the 200 improvements are already in the US version.

After using it for a while, I can say the scheduling capability is well done and nicely thought out. It covers the options you want, is easy to disable, and behaves as expected.

Overall, a minor-but-nice improvement over the EL520. If you need scheduling, I suggest sourcing a US charger on eBay (well under $100, if you can find one, as opposed to the ridiculous list price of $300) and importing… who knows when—or if—Electrolux will bring these improvements to the US.

It’s been difficult, these last few months, to find the time necessary to work on netTunes and launchTunes while working on/supporting SuperDuper! and navigating the waters of decline with Ketzl.

netTunes was pretty much done back in April, save for some online store issues—I want to support PayPal, and that means moving to the new “Embedded Web Store” (which is also required for Universal apps). Doing that involves doing some new graphics/layout for the custom pages.

launchTunes, though, was harder—ironic, since it’s a much simpler program. The big problem was that launchTunes hadn’t yet been converted to a drag-and-drop install, and I’d done some work extending its capabilities that hadn’t been completed.

Plus, I’d learned more about Cocoa in the few years since its original release (it’s never been updated)… so, it was time to throw out the old and rewrite the whole thing. Which, given the time constraints, took longer in calendar time than anticipated.

I’m happy to say that I managed to finish up the main launchTunes application, with its auto-install capabilities, while flying back from Chicago the other day. Everything tested out great, so now I’m working on the two apps’ graphics and online store stuff.

Once that’s done, the Universal versions of netTunes and launchTunes will finally get into your hands. Sorry it’s taken so long!

So, before I left, I noticed two things wrong with the server.

1. Part of Mint wasn’t working right because I hadn’t configured curl support into php4
2. CAPTCHAs weren’t working with Expression Engine because I’d left out gd

Unfortunately, I didn’t have enough time to fix these (I didn’t want to risk it, frankly), so I disabled CAPTCHAs on the blog and ignored the failing FreshView plugin.

It didn’t take long for the ‘bots to hit my comments section. Now, I hate CAPTCHAs, too, but I hate spending a significant part of my day deleting comment spam more. So, today, I fixed both problems.

Quite easy, actually. First, I tried to do it in one step:

sudo port install -v -c php4 +mysql5 +server +apache2 +imap +macosx +darwin_8 +gd +curl

But not so much. There was a differently configured php4 active—the one I’d previously built. Fortunately, easy to fix with a deactivate and then an install:

sudo port deactivate php4
sudo port install -v -c php4 +mysql5 +server +apache2 +imap +macosx +darwin_8 +gd +curl

And we’re done! After all the previous pain, positively easy.

Well, the weekend went well, the couple whose wedding we attended are happy and relieved to be done, and Lee and Mike managed to handle Ketzl with grace and aplomb.

We were a bit worried about K’s “bathroom routine” so, to make it easier for them to express K’s bladder, we put her on two drugs specifically designed for this use. This worked a bit too well—while Ketzl was able to control herself inside, as soon as she hit the great outdoors, the flood began. I think their porch stairs got a good dose: sorry about that!

Happily, their dog Red was very accepting of the intruder and got along swimmingly with the old girl.

Support was a bit of a bear—don’t you guys take holiday weekends off?—but a few hours a day, plus occasional check-ins via mobile phone browser & email—kept things moving. Hopefully, none of you who needed help felt slighted.

So, good news all around. Congratulations to Ben & Irene, and thanks to Lee & Mike (& Red) for making our attendance possible.

Zabeth and I are off to Chicago for a friend’s wedding, which means Shirt Pocket support is going to be a bit slow until the 5th: sorry to all those who are waiting more than a few minutes for answers…

This is actually the first time in nearly two years that Z and I have been able to go anywhere together. Z’s Vet School obligations and my own business get in the way, of course, but—for the most part—our fuzzy family member Ketzl has been the constraint: taking care of her is just too much to ask of anyone.

Zabeth’s classmate Lee, though, was kind enough to take her for a few days while we’re gone, and since this isn’t an entirely frivolous trip, we’re off. Lee has really gone well beyond the call of duty here, and we’re truly grateful to her and her boyfriend for their willingness to help us out—if they’re reading this, thanks! I hope we can make it up to you somehow.

Wish us, Lee, and Ketzl luck!

I’ve spent the last few days migrating Shirt Pocket from Panther Server to Tiger Server, and while it mostly went well, it could have gone smoother. Much smoother.

Hopefully, this post will help others in a similar situation. If you’re not technical, this is going to be oh-so-much gobbledegook: sorry.

The Basic Problem
Unlike upgrading from the non-Server version of Panther and TIger, Server doesn’t have any nice “Upgrade”, “archive and install”, etc. Instead, it’s a lovely clean install every time. And by lovely, I mean not lovely.

Of course, I really couldn’t bring the main server down and do some sort of multi-day transfer, so I set upon a plan. Namely, I’d take the brand new drive I planned to use (a Maxtor MaxLine III 300GB SATA), connect it to a WiebeTech SATA Dock, plug that into a G5 iMac that I usually use for testing, and install Server to it.

Which I did. Bringing up the bare-bones server configuration, adding users, etc was the easy part.

Things I Couldn’t Leave Behind
On my server, I run:

• Apache
• Kerio Mail Server
• FogBUGZ (for bug tracking, customer support, etc)
• vBulletin (our Forums)
• Expression Engine (this blog)
• Mint

Simple enough. But not. Many of the above won’t run out-of-the-box on Panther or Tiger Server. You need updated versions of PHP, extensions for Apache, updated MySQL—a whole bunch of stuff.

Prepackaged goods
Previously, I’d used Marc Linyage’s installer packages to get most of this running, and while that worked OK, my Apache server was literally crashing every few seconds due to some weird bug. And, there was no real way to update his installs until he did. I was stuck with stuff well past its update-by date.

Darwin Ports to the Rescue
So, instead, I decided to use Darwin Ports to install PHP4 (FogBUGZ doesn’t support PHP5), Apache 2.2, and MySQL 5, all with the options needed to support the above.

For those of you used to GUIs (yes, I know there are various DP GUIs), Darwin Ports would be a bit of a shock. It’s command-line only, and while it does a whole lot of great stuff, it’s a bit, well, clunky. But, I was happy to see that the PHP4 port had what I needed in it.

The Easy(ish) Part
After installing XCode and the developer tools, I installed the latest version of Darwin Ports and got to work. From what I could determine, the command to use was:

sudo port install -v -c php4 +mysql5 +server +apache2 +imap +macosx +darwin_8

That one command downloaded the sources necessary to build MySQL, PHP4, Apache 2, tweaked it to support OSX and Tiger, built the whole thing, and installed it all into the default Darwin Ports folder, /opt/local.

Configuring the Configuration
The first real challenge was getting Apache 2 configured for the Shirt Pocket site. While Apache 2 is mostly backward compatible with Apache 1.3, the httpd.conf syntax is a bit different, especially with regard to extensions. And, Apple’s Admin Console GUI doesn’t do Apache 2. So, I had a few hours’ work tweaking it to support our various realms, virtual servers and sites.

Similarly, I had to hand-configure MySQL 5 and PHP4 to add the various options we needed, including support for Kerio’s sendmail and the like.

An hour or two later, and a few port uninstall and reinstalls later (note that the PHP4 port uninstall/clean leaves bits of the install in its folders, and won’t install properly until you clean those up by hand), I had this ready to go.

Basics Are Go!
After testing things with apachectl and verifying the syntax, I used launchctl (the launchd control CLI), to add the automatically created launchd plists (nice, Darwin Ports!) into launchd and restarted. To my amazement, basic stuff was up and going.

Copying in the Site
So—next, I had to bring the data over from the other server. First, I used scp to copy over the web site files, including the installed copy of vBulletin, Expression Engine and Mint. The site seemed to work fine, although I’d neglected to create our Apache authenticated users. Once fixed, that worked too.

Welcome to The Suck
For FogBUGZ, I did a new install, and here things started to go way, way, downhill.

FogBUGZ is highly dependent on the configuration. In fact, since they recommend the Linyage packages, they’ve hardwired their paths to where he installs things in the /usr/local hierarchy and they also assume that httpd.conf is in /etc. My stuff was in /opt/local, in places FogBUGZ never knew to look, so FogBUGZ wouldn’t even install. Arrgh!

FogBUGZ support was kind enough to provide me with a rough list of prerequisites and checks they were doing, so I spent some time faking it out. After a lot of starts and stops, I determined I had to:

• Symlink /usr/bin/php to /opt/local/bin/php4
• Symlink /usr/local/bin to /opt/local/bin
• Create /usr/local/php
• Symlink /usr/local/php/bin to /opt/local/lib/php4
• Create /usr/local/php/lib
• Symlink php.ini to /opt/local/etc/php.ini

Starting to Run
Once that was done, FogBUGZ was able to install. After install, I had to tweak some configuration files:

• Copy the added fogbugz.conf from /etc/httpd/httpd.conf into /opt/local/apache2/conf/httpd.conf
• Adjust various settings in php.ini

FogBUGZ mostly came up at this point, but additional installs of PEAR components were needed (including one that FogBUGZ didn’t prompt for). But Darwin Ports’ PEAR install is split over two folders, with some components in /opt/local/lib/php4, and some in /opt/local/share/pear. So, I had to change the php.ini to add the additional include_path.

I’m Bugged
Bingo! While that took a number of hours to figure out and tweak, FogBUGZ was now up, running with a totally empty database.

So, next, data migration. Under Panther, all of the above had data in MySQL 4 in various databases and tables. Normally I’d use mysqldump, copy the files across and import them, but Bruce—who’d recently migrated machines himself—suggested using ssh to run it in “one pass”. I decided to dry-run most of the data to make sure everything worked well, while leaving the main system up (which, of course, meant the data would have to be reimported later on).

For example, to migrate the vBulletin forums, I needed to create an empty database on the new machine with the same name. Then, I ran (as root):

ssh g5server /usr/bin/mysqldump --user the-database-user --password=the-password --add-drop-table the-forum-database | /opt/local/bin/mysql5 -u the-database-user -pthe-password the-forum-database

One authentication and a few minutes later, vBulletin’s data was moved over and… vBulletin came right up!

Repeated for Expression Engine: bingo!

And, mint: no problem at all.

So, the technique worked, and the configuration seemed fine: things were up and running pretty well.

TTL - Way Long
Although FogBUGZ could be done the same way, it was going to take a lot longer because its database was absolutely huge (it takes over 14 hours to rebuild). And Mail wasn’t MySQL based, but needed to have its stuff copied across.

FogBUGZ cases come in through mail, so by turning FogBUGZ off and handling all the support through email (which means I can’t refer to earlier cases or do other things that makes this job possible) I could migrate its data. I started that on Tuesday at about 6pm.

26 hours later, it was done, and—happily—it came right up and seemed to work.

At this point, I turned off the forums and blog and pulled them over to the new server, using the same technique as before.

Goodbye Cruel Spam
Next up: Kerio Mail Server. That was going to take a while, but there was no getting around it. I installed a fresh copy of the latest Kerio release, turned it off on the real server, which would cause any inbound mail to get delayed, but would queue on hop out, and used tar w/ssh to get things across:

ssh g5server “( cd /usr/local/kerio/mailserver && tar cf - Store )” | (cd /usr/local/kerio/mailserver && tar xvpf - )

It took a few hours (lots of mail), but all the content came across, and I used scp to copy the other few individual files that needed to migrate over.

Fear of Commitment
The moment of Truth. I turned off apache and took down Shirt Pocket. Then, I quickly re-imported mint (to bring any stats I missed), and powered off the iMac and the real server.

Chatter Chatter
I love the way drives go into the G5. It’s so accessible, and so easy (OK, so drive A is a pain because you have to remove drive B, but still—great stuff), and this was as easy as as pie. Old drive out, new drive in, close up, power on and… nothing.

It just wouldn’t boot.

Oh crap.

Beginning to Panic, and not in the WWPD sense
At this point, the system was headless, so I had no way of seeing what was going on, so I quickly powered off, snagged a monitor and keyboard, and plugged everything in, being careful to smear flopsweat all over each critical connection.

First, I option-booted and… the drive was present, but Open Firmware never released the “Watch” cursor. It just sat there, spinning and taunting. Never moved on, fans started to spin faster and faster since there was no OS and… powered off again.

Tried resetting PRAM and using Open Firmware to reset both PRAM and NVRAM: no go. Removed the B-drive (which wasn’t totally necessary): no.

Clock’s ticking. Server’s down. Customer running away from company that looks like it’s gone out of business.

Mommy!
OK, deep breath, shot of espresso, think. Open Firmware’s probing for devices. What else is connected? AH! There’s a FireWire drive connected. For some reason, it’s decided to lock the bus at this exact moment. Bastard! Powered that puppy off, disconnect, start the boot again and…

and…

comes up! Login window! W00t!

The Woods are Lovely, Dark and Deep
I quickly logged in, and brought up the Server Admin to turn on DNS and make sure everything was OK. And I was greeted by something really, really bad.

Server had decided that my serial number had “already been registered”. It had shut down all services. Nothing was running. My totally legally purchased serial number, my only serial number, my $500 serial number, this isn’t-even-a developer-license-serial-number-I-bought-the-damn-thing-at-the-Apple-Store, was “already registered”. On the iMac. When I did the install to make this easier.

You Bastards!
But that server was not running? I had only installed it to this one drive. That drive was moved from machine to machine. It should be fine!

But it’s not. I’m totally, completely screwed. It’s way after hours, I can’t get the server running, Apple’s not there. I can’t go back to Panther—I’d need to spend another few days re-importing all that data.

They must be playing some Bonjour games to have that serial number float around the network in DNS or something.

Crap. Crapcrapcrap!

Bare to the Bone
There was only one thing I could think to do: whine. Rich Siegel was online, even though it was late, so I popped a message to him so he could laugh at me and my serverless foolishness. (Not that he needs an excuse to laugh at me.)

Which, of course, he did, but then—after wiping tears of joy from his eyes—pulled a rabbit out of a hat. He had a key! I could use that temporarily, until I could get whatever magic incantation I needed to get my real key working!

I typed in the key furiously (pun intended) and… phew. Server up.

It’s alive! Alive! Shirt Pocket Lives!

Cleanup
After I lowered the server down from the opening in the roof, and carefully unwrapped its bandages, things were looking pretty good. Everything was up. Everything was working. The mail was back up, running, and (alas) flooding in.

Boing!
And bouncing. Almost everything. For some reason, mail from the Forums and from FogBUGZ wasn’t going through my relay server. Instead, it was being sent directly from my server, which shouldn’t be happening.

And everything indicated it was going through Postfix. Which I wasn’t using: I had explicitly set things up to use Kerio. But, on examination of the Kerio logs, it wasn’t getting some messages. Postfix was.

You Lie Like Dog
So, I looked again in the Server Admin and… mail was off. SMTP was off. All that stuff was handled by Kerio, or should have been. But wasn’t. For some reason, Postfix was snagging stuff, even though it was supposedly off.

I messed around for a while, but at this point it was 2am, and I was exhausted. It’d been a few days since I started the Migration Adventure, and I wasn’t thinking clearly any more, so… sleep.

Coffee Achiever!
Next morning, caffeinated and slightly more awake. Same problem. But, an idea.

I took a look at the daemons in launchd and, sure enough, org.postfix.master was in there, even though Mail was off in Server Admin. It was set up to run on demand, when anything hit port 25, but—and here’s the weird part—only from localhost. So, when FogBUGZ sent to localhost:25, it triggered Postfix, which was being used instead of Kerio, on the loopback.

Man. OK: so, disable Postfix. Some tweaking later I found the trick to getting it unloaded for good:

launchctl unload -w /System/Library/LaunchDaemons/org.postfix.master

A quick system restart, and victory.

And so, we’re back up. Sorry for any bounced mail, missing site, vanished forums that happened during the process. Hopefully, it won’t happen again for a while.