Configure Software iSCSI Load-balance Multipathing to vSphere Datastores

VMware vSphere 4 gives us the ability to actively use multiple iSCSI paths to reach a single LUN. You will need to use vSphere Client, your iSCSI storage management tools, and an ESX command-line interface (such as ssh, RCLI or vSphere Management Assistant) to get it working. This procedure can be used to add up to eight iSCSI paths per datastore, provided each path uses a unique physical NIC and that each physical NIC has a corresponding NIC on the iSCSI SAN side. In other words, setting up more paths on the VMware side than your iSCSI SAN can actually accommodate would be pointless.

This tutorial assumes that you are familiar with vSphere Client and can find your way around. Read on beyond the break.

Create a Leopard to Snow Leopard Upgrade NetInstall Image

UPDATE (10/15/09): I’ve gotten mixed results from some individuals who’ve tried the workaround with 10.6.1. I’m working on and testing an extended workaround that should fix the problem for everyone. Look for that info soon.

— — — —

UPDATE (10/13/09): I’ve updated the article (and posted it as new) with a workaround for the issue that prevented additional packages from being installed. I submitted the bug that this workaround avoids to Apple — id #7247968 — on September 23, 2009.

— — — —

This post is a Snow Leopard update to a process I wrote about when Leopard (10.5) came out. This post will tell you how to create a NetInstall image that will upgrade a Mac running Leopard to the latest version of Snow Leopard in one step. It will also work to upgrade a Mac running Tiger to the latest version of Snow Leopard.

What You’ll Need

1. A 10.6 software license, or individual retail copy of Leopard, for each computer you are upgrading from Leopard or Tiger to Snow Leopard — this is easily overlooked, so let’s keep things legal.
2. A read/write disk image (.dmg) of a Mac OS X v10.6 Snow Leopard Install DVD (i.e. not shipped with a computer). You’ll convert this to read-only after making a small modification to one of the files.
3. One build computer running Snow Leopard with the latest version of Snow Leopard’s Server Admin Tools installed (10.6.0 as of this writing).
4. A computer running OS X Server providing NetBoot services — Tiger, Leopard, and Snow Leopard Server will all work
5. A copy of the OSUpgrade.pkg found on the Install DVD at /Volumes/Mac OS X Install DVD/System/Installation/Packages. This is a hidden folder, so use Go to Folder from the Finder’s Go menu to reveal it.
6. Optional: the latest Combo Updater — 10.6.1 as of this writing. (Yes, it’s not a Combo Update, but only because it’s a .1 updater.)

I. Make a Read-Write Installer Disk Image

As of Mac OS 10.5.7, there has been a bug in in the OS installer that prevents you from successfully adding sizeable packages to your System Image Utility. Fortunately, there is a workaround. I’ll walk you through how to implement it here.

1. Insert the OS Install DVD into your drive and then launch Disk Utility (found in /Applications/Utilities).
2. Select the Install DVD from the list on the left, then click the New Image icon in the toolbar.
3. Rename the image if you like, choose a location to save it, and change the Image Format option to read/write. Click Save to create the read/write disk image.
4. When the disk image is done, eject the DVD, select the disk image on the left side of the window, and click the Open icon in the toolbar to mount it.

II. Modify the Disk Image

The reason we need to modify the disk image is because, since 10.5.7, the System Image Utility can’t properly handle additional packages that you add into your workflow. During startup — whether from DVD or via NetBoot — the installer creates a RAM disk to hold the contents of the /private/var directory. This is where additional packages get placed. The RAM disk size specified by Apple’s installer is too small to hold anything buy payloadless packages. The following instructions walk you through changing the size of this RAM disk to better accommodate additional packages.

1. Open Terminal, navigate to the /etc/ directory on the disk image and open the rc.cdrom file in your favorite text editor. This file is marked as read-only, so I like to use vi as it allows you to easily write to read-only files. This command will open the file in vi:
   vi /Volumes/Mac\ OS\ X\ Install\ DVD/etc/rc.cdrom
2. The line you’ll need to modify is ‘RAMDisk /var/run 1024’. Change 1024 to 1048576 and force vi to write the read-only file. 1048576 is equivalent to 512mb — we’re specifying sectors, which are 512 bytes each — which should be sufficient for any additional packages you want to include in your upgrade NetInstall image. If your vi skills are rusty or missing, follow these step to make this change.
   1. Use the arrow keys to move the cursor down to the ‘RAMDisk /var/run 1024’ line and onto the 1 in 1024.
   2. Type cw — short for “change word”. The 1024 will disappear and vi will enter insert mode, which allows you to add characters to the file.
   3. Type 1048576
   4. Press the Esc key to exit insert mode.
   5. Type :x! to force the read-only file to be saved and exit vi.
3. Exit Terminal.

III. Convert the Disk Image to Read-only/Compressed

1. Go back to Disk Utility, select the mounted disk image on the left side of the window and click the Unmount icon in the toolbar.
2. Select the disk image and click the Convert icon in the toolbar.
3. Choose compressed from the Image Format menu. Rename the disk image so you know it’s the one you’ve modified, or save it in a different location from your read/write image. Click Save to begin the conversion.
4. Exit Disk Utility after the conversion is complete.

Keep this disk image in a safe place as you’ll be able to re-use it when you want to create an updated NetInstall image — say when the 10.6.2 Combo Update is released.

IV. Build your NetInstall Image

If you haven’t yet made a DMG from the Mac OS X v10.6 Leopard Install DVD, do it now. (I use Disk Utility to create a default compressed image.) Mount the disk image and then launch the System Image Utility found in /Applications/Server. The program will auto-detect the mounted disk image as a source and select the NetInstall Image build option.

Click Customize at the bottom of the window to bring up the workflow assembly interface.

The idea here is to drag and drop the components you want from the Automator Library window to the workflow area. The Define Image Source item is added for you at the top, with the Create Image item below it. We’re going to build the simplest possible custom upgrade image, so we’ll add only one more item to our workflow.

First, select the Add Packages and Post-Install Scripts item and drag it over in between the two existing items. Now go to the Finder and locate the OSUpgrade package. and the latest Combo Updater for Snow Leopard. You’ll have to grab the package from the mounted update dmg. (10.6.1 isn’t a combo update since there are no previous updates, but it will work just fine.) Drag the package into the list area of the Add Packages and Post-Install Scripts item.

Then, in the Create Image item at the bottom of your workflow, make sure the NetInstall option is selected in the Type section. Click the Save To drop down and select Other. Select your boot drive from the Devices list in the sidebar and click Open. (You can save directly to a network volume if you want to save yourself a step later, but I like to save mine locally so a network hiccup won’t interrupt the image build process.)

You can probably ignore the Installed Volume field since we’re working on an upgrade, but you can supply the standard volume name of the Macs you manage here. In the Image Name field, enter a descriptive name. In the Network Disk field, enter a name that you’d like the NetInstall set to be advertised as on the network. (If your network is set up to advertise your NetBoot server across the network, your NetBoot/NetInstall volumes will appear in the Startup Disk preferences pane of all the Macs on your network.) Enter a Description if you wish, and then enter a unique Index number. Choose a number between 1-4095 if you have a single NetBoot server. Choose a number between 4096-65335 if you have more than one NetBoot server in a load-balanced configuration.

Now click Save to save the workflow. When a new Combo Updater comes out later, you’ll be able to use this workflow and just drag that package in — removing any older Combo Updaters, of course.

To build the image, click Run. In a few moments, you’ll be asked for your admin credentials, after which the build process will begin. If you want to track the progress, go to the View menu and select Show Log. This is a relatively quick build, so hang around unless you really need a coffee. It should take well under 30 minutes, and as little as 10 with recent hardware.

It’s done! Now what?

Ok, so your NetInstall image was created successfully. All that’s left to do now is to copy it to your server’s (or servers’) NetBootSPx share(s). (I have mine shared via AFP, but your environment may be different.) After you’ve transferred it, use Server Admin to enable the new NetBoot image. Assuming your network is set up in a NetBoot-friendly way, you should now be able to upgrade any Mac under your control from Leopard (or Tiger) to Snow Leopard by booting it with this NetInstall image.

Don’t forget to test, test, test before using this in production.

Open a Root Finder Window in Snow Leopard

This is a Snow Leopard update to an existing post about the same topic. Yes, you can still open a root Finder window in Snow Leopard, but there is an extra step required.

First, run the following command in Terminal and then enter your password:

sudo /System/Library/CoreServices/Finder.app/Contents/MacOS/Finder

Next, click on an empty spot on your desktop — not in an existing Finder window. Now, type Command-N or select New Finder Window from the File menu. A new Finder window resembling the following should open:

You can see that it opens up to the root user’s home. Use this window to navigate anywhere you like and make the changes you need. Keep in mind that you can do just as much damage with this as you can in the Terminal as root.

To end your root Finder session, go back to the Terminal window and hit ^C.

Quirks to be Mindful of

• You won’t be able to interact with any files you might have on your desktop, as those belong to your logged-in user account and root’s desktop is currently (and transparently) sitting on top of it.
• If you take any screenshots, they will be owned by the logged-in user and you’ll need to navigate to them via your root Finder window.
• If you attempt to open/double-click a file which requires root access to read, the corresponding application will open as the logged-in user and the file will fail to open. To get around this, you can launch the app’s /Contents/MacOS executable as root and open the file from within the app.

Get a Twitter Account’s Registration Date via Snow Leopard Service

Creating services in Snow Leopard is all the rage lately. So is Twitter. This tutorial brings both of them together and should serve as a fine example of just how slick and useful services in Snow Leopard can be.

Recursively Find/Replace Inside Files Within a Directory

We recently had to change a handful of usernames in LDAP due to a merging of resources. This was a relatively painless process, but since some services use static authorization files to grant access, some manual post-processing was necessary. The script at the end of this post is something I came up with to deal with updating the subversion auth_files. It’s a bash script that uses a couple useful tricks:

tree -ifF --noreport /path/to/dir/ | grep -v '/$'

• The tree command normally prints out an ascii-graphical representation of the file structure rooted in the given path, recursively. The ‘-i’ option tells it not to display the graphics. The ‘-f’ option prints out the full path to each item. The ‘-F’ adds file-type indicators to the end of file names, which I’m using here so I can filter out directories from the list using an inverse grep.
• The output of the tree command is piped to grep. The ‘-v’ option activates inverse grep, and the ‘/$’ regex will match trailing slashes. This grep will match all lines not ending in ‘/’.
• The tree command is not standard on all flavors/versions of *nix. It’s missing on OS X, for example.

perl -p -i -e 's|before|after|[ig]' file

• This perl command will edit a file in-place, replacing occurrences of “before” with “after”.
• Adding an i to the end of the substitution string makes it a case insensitive substitution.
• Adding a g makes the command replace all instances, a.k.a. global, instead of just the first instance.

Why perl instead of sed for in-place edits?

Not all versions of sed allow in-place edits, especially older ones, so perl is the more universal option. If you know your sed can do in-place edits (check the man page for the ‘-i’ option), then you can replace the perl line in the script below with this:

sed -i'' -e "s|$1|$2|g" $afile

Whether you choose to use perl or sed, you must remember to double-quote the substitution string so bash expands the variables and hands the values off to sed/perl. Using single quotes here would result in sed/perl looking for a literal ‘$1′ to replace with a literal ‘$2′.

The Script

This code can easily be repurposed for other tasks, but I present it here as I wrote it for the subversion auth_files purpose. (I named it “auth_find”.)

#!/bin/bash

workpath=/opt/auth_files/
outfile=/root/auth_find-$1

# friendly usage funtion, called if no argument is supplied
usage ()
{
    echo ""
    echo "Usage: auth_find [username] [new-username]"
    echo ""
    echo "This script recursively searches subversion's /opt/auth_files/ directory for"
    echo "the supplied username and returns a list of files that contain it. If a second"
    echo "username is supplied all instances of the first will be replaced with the second."
    echo ""
    echo "Output is sent to both STDOUT and /root/auth_find-username."
    echo ""
    exit 1
}

if [ $# == 1 ]; then    # do this block if one argument is given
    echo "Results:"
    for afile in $(tree -ifF --noreport $workpath | grep -v '/$'); do
        if [ -n "$(grep "^$1 " $afile)" ]; then
            echo "$afile" | tee -a $outfile
        fi
    done
else
    if [ $# == 2 ]; then    # do this block if two arguments are given
        echo "Now replacing occurrences of '$1' with '$2' in the following files:"
        for afile in $(tree -ifF --noreport $workpath | grep -v '/$'); do
            if [ -n "$(grep "^$1 " $afile)" ]; then
                echo "$afile" | tee -a $outfile-CHANGED
                perl -p -i -e "s|$1|$2|g" $afile
            fi
        done
    else    # show usage if incorrect number of arguments given
        usage
    fi
fi
#!/bin/bash
outfile=/root/auth_find-$1

UPDATE (8/31/09): Added “why perl instead of sed” section in response to comment.

Can I boot Snow Leopard in 64-bit mode?

UPDATE: Please read Update 2 at the bottom of this post before using a 64-bit kernel as your default.

---

With Snow Leopard making its appearance this Friday, August 28, 2009, some people may be wondering whether they’ll be able to boot their Macs in 64-bit mode. Only Intel Xserves will boot this way by default. If you want to boot your desktop or mobile Mac in 64-bit mode, you’ll need to take some additional steps. The first is checking to see if your Mac has a 64-bit-capable EFI. If the output of the following command is EFI64, you’re good. If not, you’re out of luck.

    ioreg -l -p IODeviceTree | awk -F'"' '/firmware-abi/{print $4}'

Once you’ve verified it’s possible, you have a couple options for making your Mac boot into 64-bit mode. I’d try them in this order. First, to affect the current boot only, hold down the ‘6′ and ‘4′ keys during bootup. Once you’ve verified it works and are comfortable with it, you can make the change permanent by adding an ‘arch=x86_64′ boot flag to your com.apple.Boot.plist, like so:

    sudo defaults write /Library/Preferences/SystemConfiguration/com.apple.Boot 'Kernel Flags' 'arch=x86_64'

---

UPDATE 1 (8/28/09): Apple has a couple new (and one older) knowledge-base articles pertaining to this topic.

1. Mac OS X Server v10.6: Macs that use the 64-bit kernel
2. Mac OS X Server v10.6: Starting up with the 32-bit or 64-bit kernel
3. How to tell if your Intel-based Mac has a 32-bit or 64-bit processor

---

UPDATE 2 (8/29/09): This post has received quite a few hits, so I now feel the need to include some educational material about why Apple chose to make Snow Leopard boot with a 32-bit kernel by default.

The primary reason is for compatibility with third-party software, particularly software that requires kernel extensions. Probably the most widely know examples of software that depends upon kernel extensions, or kexts, are VMware Fusion and Parallels. If you use these to run Windows or Linux on your Mac, you’ll want to keep using a 32-bit kernel. Virtualization software needs direct access to the hardware normally controlled by the kernel (CPU, RAM, Disk) in order to “fool” operating systems into thinking they’re installed on “real” computers. The kernel extensions allow them to do this.

Kexts must be written specifically for 32-bit or 64-bit kernels. They are not interchangeable. Applications, on the other hand, can run at 64-bit even if the kernel is 32-bit. As far as your 64-bit CPU is concerned, the kernel is just another application. It’s a very important application — in the sense that it is code that is executed on a processor — whose job it is to arbitrate demands on the system’s resources. Most applications don’t have direct access to the CPU, RAM, or other physical devices, but make requests of the kernel instead.

---

UPDATE 3 (9/1/09): John Siracusa’s new article on Snow Leopard was posted today. Then entire thing is great reading, but I’m linking to the section that addresses 64-bit vs 32-bit here.

Boost VirtualBox disk I/O for Windows VMs

I picked up a VirtualBox Windows VM optimization tip from the MacEnterprise mailing list this morning, supplied by Yadin Flammer. Yadin mentioned that switching your Windows VM’s disk type from the default IDE to SATA and using the Intel Matrix Storage drivers results in faster hardware emulation. I decided to verify this claim by collecting some before and after I/O data. I have a Windows XP VM, but this should apply to all versions of Windows from 2000 onward, both server and client.

I used the freely-available Iometer to gather my disk I/O data. Both the before and after tests were run for 5 minutes on an ~ 3GB test file using the All-In-One test suite. Here are the results.

As you can see, the data clearly shows a slight increase in performance. Using SATA is actually recommended by Sun, as well. On this page, they say,

Like a real SATA controller, VirtualBox’s virtual SATA controller operates faster and also consumes less CPU resources than the virtual IDE controller. Also, this allows you to connect more than three virtual hard disks to the machine.

Makes sense, no? In terms of MBps, my IDE test averaged 17.925 while my SATA test averaged 18.828. Now that we know it’s better, we’ll move on to the installation and configuration procedure.

Installation and Configuration

1. Shut down your Windows VM and open its settings window (shown above).
2. Select the Hard Disks item, check Enable Additional Controller and choose SATA (AHCI) from the list.
3. Leave the Hard disk attached to the IDE Controller in the Attachments section for now, since we’ll first have to install the SATA drivers, and click OK.
4. Start up your VM again and download the Intel Matrix Storage Manager drivers. Click the link, select your Windows OS version, click Go, and then click the first download link in the Drivers section. Mine showed up as link #1. Save it to your desktop, and then install it. NOTE: If you are given a warning about not meeting the minimum installation requirements, you may need to download and install the Intel Chipset Software Installation Utility first. Follow the same download and install procedure as for the storage drivers.
5. Once the drivers are installed, shut down your Windows VM and open its settings window.
6. Select the Hard Disks item as before, but now select SATA Port 0 next to your VM’s .vdi file in the Attachments section.
7. Click OK and then start up your VM.

… And boom goes the dynamite.

Print PDFs as Postscript to an lpr Queue

I wrote a simple script recently for a user who was having trouble getting certain PDFs to print properly from his linux box (Fedora 10). I first suggested that he try converting the pdfs to ps and printing the resulting file. That worked but he found the process a bit tedious. Here’s the script I wrote to take care of the tediousness. It relies on the standard (in Fedora, at least) pdf2ps package. It should be pretty self-explanatory.

#!/bin/bash

# grab first argument as pdf filename and generate ps filename
thePDF=$1
thePS=$(echo $thePDF.ps)
queueName=$2

usage()
{
    echo ""
    echo "Usage: psprint [your pdf] [lpr queue]*"
    echo ""
    echo "This command does three things:"
    echo "  1. Converts the specified pdf file to ps"
    echo "  2. Prints the ps file to your default lpr queue *(unless you specify another queue)"
    echo "  3. Deletes the ps file"
    echo ""
    exit 1
}

if [ $# == 0 ]; then usage; fi
if [ $# == 1 ]; then
    echo "Converting $thePDF ..."
    pdf2ps "$thePDF" "$thePS"
    echo "Sending to default printer ..."
    lpr "$thePS"
    echo "Cleaning up ..."
    rm "$thePS"
    exit 1
fi
if [ $# == 2 ]; then
    echo "Converting $thePDF ..."
    pdf2ps "$thePDF" "$thePS"
    echo "Sending to $2 ..."
    lpr -P "$queueName" "$thePS"
    echo "Cleaning up ..."
    rm "$thePS"
    exit 1
fi
if [ $# > 2 ]; then usage; fi

Save the script to a location in your path (/usr/local/bin works) and you’re off.

Open a Finder Window with Root Access

UPDATE (9/28/09): Got Snow Leopard? Please see this post for updated instructions.

— — — —

It’s occasionally handy when troubleshooting a problem in OS X to have root access in the Finder without having to log out of your current session. Sure, you can do most things in the Terminal, but the GUI can be much handier for certain tasks. This is a quick-and-dirty Terminal trick to open a Finder window with root access.

Run the following command and then enter your password:

sudo /System/Library/CoreServices/Finder.app/Contents/MacOS/Finder

A new Finder window resembling the following should open:

You can see that it opens up to the root user’s home. Use this window to navigate anywhere you like and make the changes you need. Keep in mind that you can do just as much damage with this as you can in the Terminal as root.

To end your root Finder session, go back to the Terminal window and hit ^C.

Quirks to be Mindful Of

• You won’t be able to interact with any files you might have on your desktop, as those belong to your logged-in user account and root’s desktop is currently (and transparently) sitting on top of it.
• If you take any screenshots, they will be owned by the logged-in user and you’ll need to navigate to them via your root Finder window.
• If you attempt to open/double-click a file which requires root access to read, the corresponding application will open as the logged-in user and the file will fail to open. To get around this, you can launch the app’s /Contents/MacOS executable as root and open the file from within the app.

Quickly Add a Userset to Many Sun Grid Engine Queues

This will be the first (of many??) posts to spill outside the topics one would think you’d find on a site with the name “Your Mac Guy”. You’ve been warned.

Back in January my primary work responsibilities shifted from Mac servers and desktops (and all that entailed) to Linux servers and desktops and the multitude of new things that entails (at least here where I work). One of the new tasks I’ve picked up is user administration of our Sun Grid Engine (SGE) 500-node cluster. New or existing users who want to submit jobs to the cluster need to be added to custom groups or, in SGE-speak, usersets. We create usersets for each lab, so if the user is part of a lab that doesn’t currently have access to submit jobs, I need to create a new userset and add that userset to each of 16 separate queues.

That last part, adding usersets to queues, is the most tedious part. So tedious, in fact, that it forced my hand into developing a scripted solution. I likely could have found an existing script to accomplish the task for me, but then I wouldn’t have had an excuse to brush up on my 3-years dormant perl skills.