AIX Updates With Multibos

Starting with AIX* 5.3 Technology Level (TL) 3, the new multibos utility has been
provided that allows an AIX administrator to create and maintain two separate, bootable
instances of the AIX OS within the same root volume group (rootvg). This second
instance of rootvg is known as a standby Base Operating System (BOS) and is an
extremely handy tool for performing AIX TL and Service Pack (SP) updates.

Multibos lets you install, update and customize a standby instance of the AIX OS without
impacting the running and active production instance of the AIX OS. This is valuable in
environments with tight maintenance windows. Instead of requiring an outage window of
several hours to apply a new TL or SP, you’ll only need a small outage at a convenient
time to reboot the system. This may help reduce the size of the after-hours effort often
required when performing AIX updates, as all the maintenance activities can be
performed during business hours. After hours you could log in from home and reboot the
system.

Backing out from an AIX TL update is also made easier with multibos. To go back to a
previous TL, you reboot the system on the original AIX instance boot logical volume
(BLV). It’s also possible to update several AIX systems at once using multibos, which
again reduces the amount of after-hours effort required when performing AIX
maintenance activities.

Multibos is similar to an alternate disk installation. However, there are several differences
between the two methods, one of which is that there’s no need for an additional disk to
clone the rootvg. Both utilities can be used to achieve the same goal. Choose the one
that’s the best fit for your AIX environment.

Getting Started

Before attempting to use multibos, check that the prerequisites have been met. First, the
system must be running AIX 5.3 with TL3 or higher. Next, ensure that there’s enough
free space in rootvg for a copy of each BOS logical volume (LV). By default, the BOS
file systems in rootvg(/, /usr, /var, and /opt) and the BLV are copied. All other file
systems and LVs are shared between BOS instances. Check the number of free physical
partitions in rootvg (i.e., # lsvg rootvg | grep FREE). If all of the requirements can’t be
met, then a traditional update should be performed.

Ensure that you document the system and perform a mksysb before performing any
maintenance activity.
Creating a Standby BOS Instance

In the following example we’ll upgrade an AIX 5.3 system from TL4 to TL6. Before
starting, remove any previous standby BOS instance and associated log file using
“multibos –R.” This creates a clean environment and makes reviewing the log file easier.
Then preview the operation first (using “multibos –sXp”) and review the multibos log for
any errors (i.e., “alog –of/etc/multibos/ logs/op.alog”).

After a successful preview, create the standby BOS instance. This may take several
minutes and should create a new LV and file system for each BOS copy. Each new LV
and file system is prefixed by bos_. You should review the log file for errors. See Code
Sample 1.

We can now start a multibos shell and verify the standby BOS and its current TL. See
Code Sample 2.

Updating and Booting the Standby BOS

Now we can apply TL6 to the standby BOS. The TL6 lppsource is mounted from our
Network Installation Manager (NIM) master. Perform a preview operation and then
execute the actual update to the standby instance. Again, checking the log file for any
issues:

# mount nimmast:/export/lpp_source/lpp_sourceaix530603 /mnt

# multibos -Xacp -l /mnt
# multibos -Xac -l /mnt

We’ve now successfully applied TL6 to the standby BOS. Start a multibos shell again
and verify that the standby BOS TL has been updated:

# multibos –S
...
# oslevel -s
5300-06-03-0732
# exit

We’re now ready to boot the standby BOS, which can be performed some time after the
standby instance has been updated. The BOS may be upgraded during the day, but the
system may be restarted after hours depending on the available outage windows for this
system. Thus far, all maintenance activities have been performed on a live production
system without the need for an outage. Before rebooting, verify the bootlist and ensure
that the BLV is set to the standby BOS (bos_hd5).

When booting a standby BOS instance you can check that the correct BLV is being used
by comparing the output from the bootlist command with the output from the “Welcome
to AIX” banner. You should record the output from the bootlist command, as it may
come in handy if issues arise later. In Code Sample 3, the BLV hd5 relates to device 2.
Code Sample 4 displays what should appear during system startup.

Reboot the system now with the “shutdown –Fr” command. After the reboot, confirm the
TL level via the “oslevel –r” command. You can also verify which BLV the system
booted from with the “bootinfo –v” command.

The multibos LVs and file systems are now open and mounted. The original file systems
are renamed with a bos_ prefix but their original LV names are preserved.

If you need to back out of the update and return to the previous TL, all that’s required is
to boot the previous (“original”) BOS instance. Set the bootlist and verify that the BLV is
set to the previous BOS instance (hd5):

# bootlist -m normal hdisk0 blv=hd5 hdisk0 blv=bos_hd5

# bootlist -m normal -o
hdisk0 blv=hd5
hdisk0 blv=bos_hd5

Now reboot the system and confirm that it’s running at the previous TL.

It’s worth mentioning that if the system fails to boot the standby BOS, you can attempt to
start the system in maintenance mode (e.g., via NIM) and change the bootlist to point at
the alternate BLV. If required—in rare cases—you can also boot the system from the
open firmware prompt. You can reach the open firmware prompt by hitting the 8 key
while booting. From the boot prompt you can specify the desired boot device, as shown
here:

0> boot /vdevice/v scsi@30000004/disk@8100000000000000:4 |

/vdevice -
Virtual I/O Bus
/v-scsi@30000004 -
Virtual I/O SCSI Adapter
/disk@8100000000000000:4
- Virtual I/O SCSI Disk Device

Multibos and dsh

It’s possible to perform multibos operations on several systems at once using multibos
and the dsh command. The dsh command allows you to execute commands on multiple
hosts in parallel. The example in Code Sample 5 (right) will perform multibos operations
on all six hosts specified.

After the reboot, verify all of the hosts have been updated by running the “dsh oslevel –r”
and “dsh oslevel –s” commands.
All of this work (up to the point of needing to reboot) can be carried out on live
production systems without an outage.

Powerful Administration Tool

The multibos command is a powerful AIX administration tool. It has the potential to
reduce outage windows required for AIX TL maintenance activities and also alleviate
some of the after-hours effort on the part of the AIX administrator.

Resources

/usr/lpp/bos/README.multibos
http://publib.boulder.ibm.com/infocenter/systems/topic/com.ibm.aix.install/doc/insgdrf/m
ultibosutility.htm
www.ibm.com/collaboration/wiki/display/WikiPtype/Boot+Process+on+Power
publib16.boulder.ibm.com/doc_link/en_US/a_doc_lib/cmds/aixcmds2/dsh.htm