Probably

Sun have done some work in recent times with liveupgrade – the last time I looked at it, a few years back now, it was rubbish. I thought it was about time I took another look, since a lot of the updates in OpenSolaris were looking good.

The idea was to patch a Solaris 10 update 8 (10/09) machine to the most recent patch levels, whilst the machine was still up and running, going about it’s daily business. Other that the standard Solaris tools, I’d be using pca (Patch Check Advanced) to do the actual patching. The system was installed with a ZFS root, since this actually gets us some great features in LiveUpgrade (LU) – namely ZFS snapshots as boot environments (BEs).

First off, create a BE that will be patched:

solaris:~# lucreate -n patching
Checking GRUB menu...
System has findroot enabled GRUB
Analyzing system configuration.
Comparing source boot environment file systems with the file
system(s) you specified for the new boot environment. Determining which
file systems should be in the new boot environment.
Updating boot environment description database on all BEs.
Updating system configuration files.
Creating configuration for boot environment .
Source boot environment is .
Creating boot environment .
Cloning file systems from boot environment to create boot environment .
Creating snapshot for on .
Creating clone for on .
Setting canmount=noauto for in zone on .
WARNING: split filesystem file system type cannot inherit
mount point options <-> from parent filesystem file
type <-> because the two file systems have different types.
Saving existing file in top level dataset for BE as //boot/grub/menu.lst.prev.
File propagation successful
Copied GRUB menu from PBE to ABE
No entry for BE in GRUB menu
Population of boot environment successful.
Creation of boot environment successful.

If we now take a look at the ZFS filesystems we can see the ‘patching’ snapshot…

solaris:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 2.08G 5.73G 33K /rpool
rpool/ROOT 1.03G 5.73G 21K legacy
rpool/ROOT/install 1.03G 5.73G 1.03G /
rpool/ROOT/install@patching 59.5K - 1.03G -
rpool/ROOT/patching 120K 5.73G 1.03G /
rpool/dump 560M 5.73G 560M -
rpool/export 44K 5.73G 23K /export
rpool/export/home 21K 5.73G 21K /export/home
rpool/swap 512M 6.14G 100M -

Let’s see what lustatus shows us now…

solaris:~# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
install yes yes yes no -
patching yes no no yes -

So we have two boot environments.

LU has a nice feature of letting you mount a BE to do ‘work’ on it. Let’s see what’s mounted, and then mount our newly created ‘patching’ BE…

solaris:~# lumount
install on /
solaris:~# lumount patching
/.alt.patching
solaris:~# lumount
install on /
patching on /.alt.patching

So now our alternate boot environment is mounted as /.alt.patching, we can go ahead and patch it. pca supports patching to an alternative root with the -R switch, much like Solaris packaging tools…

solaris:~# pca -i -R /.alt.patching
[snip]
------------------------------------------------------------------------------
141505 04 < 07 RS- 28 SunOS 5.10_x86: ipf patch
Looking for 141505-07 (29/84)
Trying SunSolve
Trying https://sunsolve.sun.com/ (1/1)
Done
Installing 141505-07 (29/84)
Unzipping patch
Running patchadd
Done
Reboot recommended
------------------------------------------------------------------------------
[snip]
------------------------------------------------------------------------------
Download Summary: 84 total, 84 successful, 0 skipped, 0 failed
Install Summary : 84 total, 84 successful, 0 skipped, 0 failed

This could take a while.

When the patching is complete, unmount the BE and set it to be the active one on the next reboot…

solaris:~# luumount patching
solaris:~# lumount
install on /
solaris:~# luactivate patching
System has findroot enabled GRUB
Generating boot-sign, partition and slice information for PBE
Saving existing file in top level dataset for BE as //etc/bootsign.prev.
A Live Upgrade Sync operation will be performed on startup of boot environment.
Generating boot-sign for ABE Saving existing file in top level dataset for BE as //etc/bootsign.prev.
Generating partition and slice information for ABE Copied boot menu from top level dataset.
Generating multiboot menu entries for PBE.
Generating multiboot menu entries for ABE.
Disabling splashimage
Re-enabling splashimage
No more bootadm entries. Deletion of bootadm entries is complete.
GRUB menu default setting is unaffected
Done eliding bootadm entries.
**********************************************************************
The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.
**********************************************************************
In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:
1. Boot from Solaris failsafe or boot in single user mode from the Solaris
Install CD or Network.
2. Mount the Parent boot environment root slice to some directory (like /mnt). You can use the following command to mount:
mount -Fzfs /dev/dsk/c0d0s0 /mnt
3. Run utility with out any arguments from the Parent boot
environment root slice, as shown below:
/mnt/sbin/luactivate
4. luactivate, activates the previous working boot environment and
indicates the result.
5. Exit Single User mode and reboot the machine.
**********************************************************************
Modifying boot archive service
Propagating findroot GRUB for menu conversion.
File propagation successful
File propagation successful
File propagation successful
File propagation successful
Deleting stale GRUB loader from all BEs.
File deletion successful
File deletion successful
File deletion successful
Activation of boot environment successful.

Notice the message about what to do to recover the old session should the boot fail. Personally I keep a copy of that notice to hand, just in case. Evernote is particularly handy I find.

So if we now look at lustatus, we can see our patching BE is the active on reboot…

solaris:~# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
install yes yes no no -
patching yes no yes no -

So let’s go ahead and reboot at a time that suits us. When the system comes back up we can see ‘patching’ is now the active BE…

solaris:~# lustatus
Boot Environment Is Active Active Can Copy
Name Complete Now On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
install yes no no yes -
patching yes yes yes no -

And pca shows us there are no patches to be applied, so we’re up to date…

solaris:~# pca -l
Using /var/tmp/patchdiag.xref from Mar/02/10
Host: solaris (SunOS 5.10/Generic_142901-05/i386/i86pc)
List: missing (0/0)

zfs list shows us that the patching snapshot is now using up space too…

solaris:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 2.82G 5.00G 36.5K /rpool
rpool/ROOT 1.77G 5.00G 21K legacy
rpool/ROOT/install 31.3M 5.00G 1.05G /
rpool/ROOT/patching 1.74G 5.00G 1.37G /
rpool/ROOT/patching@patching 377M - 1.03G -
rpool/dump 560M 5.00G 560M -
rpool/export 44K 5.00G 23K /export
rpool/export/home 21K 5.00G 21K /export/home
rpool/swap 512M 5.40G 100M -

Using a recent Solaris 10, with ZFS root, LU and pca gives us a very realistic way of patching systems in a working, production, environment without the pain of downtime and with a workable roll back strategy.

Now, if I had time to write something to centralise this for many hosts, it would make a fantastic enterprise setup :)