10.3 to 10.4 server upgrade and mirrors

The transition from 10.3 to 10.4 in our location has been both easy and hard. For everything except our master server, the transition has been a breeze.

Here are some things to do and a couple of things to avoid. Note: this is only for OS X Server, we've seen zero problems from doing client upgrades).

We've done all of the good check items:

  • Make sure that you're running a clean version of the latest OS (10.3.9 w/the security updates)
  • Depending upon your superstitious nature, try a Repair Permissions from Disk Utility
  • Definitely try a Verify from Disk Utility
  • Shut down all services
  • Back up everything
  • Back up all configuration files (/private/etc in particular)
  • Back up all email (on mail servers) &mdash /private/var/imap /private/var/mail/spool
  • Back up all DNS information (on DNS servers) &mdash /private/var/named
  • Back up all local NetInfo information (especially if you are running DHCP, as it is mostly stored there)
  • Run Apple's installer from the CDs or DVD as appropriate
  • Use the same user name/short user name/password on the server as was there before
  • Once the setup is confirmed, verify all of your configuration files
  • Import certificates into the certificate manager and update all certificate-bearing sites and processes (saves headache next time they all expire)
  • Once everything looks good, turn the services on and verify that the services are doing the right thing Sounds like a good enough idea, and I was happy with the results on almost all of my OSX servers. Unfortunately, on our main server, I decided to stray a bit from the above suggestions and added the step of popping out half of the mirrored drives (aka "breaking the mirror"). On other operating systems and even previous OS X upgrades, I'd done this successfully, although it isn't suggested. The basic premise is that half of a mirror (pulled while the computer is turned off) is an exact, bootable copy of the server that you just had and is the fastest way to get back up and running.

Unfortunately, especially between 10.3.x and 10.4, this isn't really the case. There seem to be some "issues" between these two operating systems and their RAID drivers (AppleRAID 1 and AppleRAID 2) that give severe trouble when upgrading.

I had already upgraded on multiple systems that had currently operational AppleRAID 1 volumes and those all worked fine. However, this particular method (popping out half of the RAID under AppleRAID 1 during the 10.3 to 10.4 migration) was fatal for the array.

Upon completing the upgrade (another lengthy story that I may put up here in the future) and popping the old drive back in, we were unable to recover the raid through either the command line or the Disk Utility program.

Eventually, the problem required the creation of a new boot volume, Carbon Copy Cloning the data between the broken RAID and the new boot volume and rebooting with that to decommission the old RAID.

However, the saga didn't end there. Due to some weirdness between AppleRAID 1 and AppleRAID 2, getting the old drive to accept joining a new raid required a complete zeroing of the disk (think in terms of tens of minutes or an hour, depending on the size of disk).

All things considered, CCC to a backup firewire drive was more than sufficient and would have saved me a lot of pain.

On a related note, I also had problems converting a degraded AppleRAID 1 drive to AppleRAID 2 (which I was attempting to do in order to circumvent the first problem).

So, do what I say, not what I did...