Installing OpenSolaris on a server with a HP Smart Array Controller

I'm quite new to OpenSolaris; insofar as it's like Linux, I'm pretty comfortable, but there appear to be enough little gotchas to make the learning curve a little steeper than one might like. Take, for instance, the fact that OpenSolaris doesn't ship with the drivers for the almost-ubiquitous HP Smart Array controller. That makes for an installation hiccup that a quick Google search shows many people have found difficult. It's too bad, too: it appears that this has stymied a lot of folks in trying to install the OS. Like most things, the solution is easy, once you know how to do it. These instructions, originally, were for OpenSolaris 2008.11, but they've since been updated to work with 2009.06.

Boot to the Install CD

The first task is to boot to the OpenSolaris Live CD, which is also the install CD. If you haven't downloaded it, yet, you can get it at http://opensolaris.org

Get the HP Smart Array Driver

Download the CPQary3 driver, which includes Solaris drivers for most of the recent (and not so recent) Smart Array controllers. As of this writing, the latest version of these drivers is v2.2.0. 

Update: I've tried a few times to get the v2.0 drivers to work on the latest-generation HP servers, and I've had no luck. The v2.2.0 drivers are out, and they support the latest that HP has to offer. I've confirmed that the 2.1.0 drivers work on a BL460 G6 .

So stay away from the 2.0.0 drivers with new HP servers: they can cause some serious heartburn. The HP CPQary drivers can be found here. The /tmp filesystem has a lot of space on the live CD, so it's probably best to save the file there. 

I saved off the 1.9.2 drivers here, if that's helpful. As always, it' a better idea to go to the source (HP in this case) for files than a second-hand location, but if you can't locate it at their site, it'll remain available here.

Install the Driver

Now that we've got the drivers, we'll unpack them and go about installing them:

jack@opensolaris:/tmp$ tar -zxf *.gz
jack@opensolaris:/tmp$ ls
CPQary3-2.0.0-solaris10-i386
dbus-D43WuQsGnK
iconf_entries.363
CPQary3-2.0.0-solaris10-i386.tar.gz
dbus-EmWPHCv5Ec
ogl_select471

Just for simplicity's sake, I renamed the directory, so that it was a bit less unwieldy.

jack@opensolaris:/tmp$ mv CPQary3-2.0.0-solaris10-i386 cpqary
jack@opensolaris:/tmp$ cd cpqary
jack@opensolaris:/tmp/cpqary$ ls
CPQary3.144
 CPQary3.pkg
 LICENSE.CPQary3
 RELEASENOTES.CPQary3
CPQary3.iso
 DU
 README.CPQary3
 tools

Note that the OpenSolaris Live CD logs in with the username 'Jack,' which doesn't have much in the way of priviledges. Instead of sudo, use the pfexec script to run the commands with elevated priviledges. 

Now here is where some persistent Googling paid off. There's a bug report at the OpenSolaris site (bug #5860) where a developer suggests a step (creating an empty file in the root dir) that makes things all OK.

Note that this step continues to be necessary with OpenSolaris 2009.06 and the v2.1.0 CPQary3 drivers. So here are the rest of the steps: 

Create a file on root:

jack@opensolaris:/tmp/cpqary$ pfexec touch /ADD_DRV_IGNORE_ROOT_BASEDIR

Once we've done that, we can install the driver we downloaded:

jack@opensolaris:/tmp/cpqary$ pfexec pkgadd -d ./CPQary3.pkg 
The following packages are available:
1 CPQary3 HP Smart Array Controller Driver (i386)
2.0.0,Rev=2008.12.05.01.09
 Select package(s) you wish to process (or 'all' to process all packages).
(default: all) [?,??,q]:

Processing package instance from HP Smart Array Controller Driver(i386) 2.0.0,Rev=2008.12.05.01.09

Copyright 2008 Hewlett-Packard Development Company, L.P.
## Executing checkinstall script. Using as the package base directory.
## Processing package information.
## Processing system information. 11 package pathnames are already properly installed.
## Verifying package dependencies.
## Verifying disk space requirements. WARNING: The /usr filesystem has 0 free blocks.
The current installation requires 158 blocks, which includes a required 150 block buffer for open deleted files.
158 more blocks are needed.
WARNING: The /usr filesystem has 0 free file nodes.
The current installation requires 26 file nodes, which includes a required 25 file node buffer for temporary files.
26 more file nodes are needed.
 Do you want to continue with the installation of [y,n,?] y 
## Checking for conflicts with packages already installed. 
## Checking for setuid/setgid programs. 
This package contains scripts which will be executed with super-user permission during the process of installing this package. 
 Do you want to continue with the installation of [y,n,?] y 
 Installing HP Smart Array Controller Driver as 
## Installing part 1 of 1. 
/kernel/drv/amd64/cpqary3 
/kernel/drv/cpqary3 
/kernel/drv/cpqary3.conf 
/usr/share/man/man7d/cpqary3.7d 
ERROR: attribute verification of failed pathname does not exist [ verifying class ] 
ERROR: attribute verification of failed pathname does not exist [ verifying class ] [ verifying class ] 
## Executing postinstall script. 
Installation of partially failed.

Note a couple of things: the defaults are sufficient, and there are errors in the install. Happily, the errors are in copying the man pages, which we'll not need, at least for now (the /usr filesystem is read-only in the live CD). The good news is that the driver now is installed for the Smart Array controller.

Install OpenSolaris

Double-click on the "install OpenSolaris" icon, and you now should be able to see your drives for installation.

Troubleshooting

Reboot loop

If, after installing OpenSolaris as above, you find yourself in a reboot loop, the best thing to do in troubleshooting it is to set a boot option such that OpenSolaris will display text as it's booting, rather than a graphical progress screen. To enable this, type the letter e when presented with the GRUB boot loader menu. This will allow you to edit the boot options. You'll see a list of the steps that are used in booting OpenSolaris. First we want to get rid of the splash screen. Highlight the line that references "splashimage" and press the letter d. That will delete the image that otherwise will cover up the debugging text. Now select the kernel line (it begins with 'kernel$') and type the letter e again. If you're unfamilar with GRUB, you're getting a glimpse into how it works: basically, it's a series of commands that sets up the system and then passes control over to the operating system. It's a great system for dual booting (Windows, unbeknownst to most, uses something similar, if more mysterious). So now we've got a line that, by default, looks more or less like this:

kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS,console=graphics

We want to add verbosity and debugging to the boot options, so add -k -v to that line and remove the graphical console, such that it reads like this:

kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS -k -v

Hit ENTER, and you'll be returned to the GRUB menu, where you can hit the letter b to boot the system. Now you can see the error that is causing the system not to boot (or to reboot). In many cases, you'll see that the error is "cannot mount root" because there's something wrong with the SCSI controller driver. In this case, try the installation again, using an older version of the CPQary3 drivers.

Enabling Event Logging

HP, for some reason, turned off storage controller event logging in the latest version of the CPQary3 driver. You can read the full HP Advisory here, along with instructions on enabling the logging.