Secrets of an AIX Administrator, Part 3: Tips for Patching Your AIX Server

Nowhere is the truism "the only constant thing is change" more relevant than in the IT world. We have to deal with the constant presence of external threats such as exploits and vulnerabilities that must be removed to prevent servers from being compromised. Then, too, software is continually updated as new tools are released and features are added to existing software and OSs. So as technology constantly evolves, keeping AIX servers up to date with the latest versions of the OS is crucial. In this article, we’ll look at how to patch AIX servers and maintain software on them.

Breaking Down the OS

Before we delve into the actual topic of patching, I want to back up a bit and discuss AIX's versioning structure. AIX has one of the easiest structures for keeping track of and understanding the levels of software on a server. Unlike Microsoft Windows, with its non-intuitive version numbers—for example, Windows 98 was released long before Windows 7—or an OS whose patch levels aren’t easily discerned when you look at the individual pieces of the OS at a glance (say, when watching a Solaris patch cluster as it runs), AIX has a simple way of iterating each version of its OS and components.

The IBM AIX OS comprises four main levels: version, release, technology level (TL), and service pack (SP). The version and release numbers denote the main levels of AIX and typically change as major features or hardware enhancements are released. The next level under Release is TL; TLs are groupings of large quantities of OS filesets that contain new functions or features. Finally, under TLs are SPs, which are more minor revisions (i.e., fixes) within each TL. You can view the levels installed on your system by running the oslevel -s command.

Likewise, AIX filesets generally reflect the OS level. You can obtain the levels for a particular fileset by running the lslpp -l command against the fileset; the four numbers returned in the command output generally reflect the version, release, TL, and SP pretty well. However, other pieces of software (i.e., DB2) often don’t have any relationship to the level of the AIX OS.

Commit-Apply-Commit

The first step to maintaining the AIX OS on servers is to understand the difference between a committed and an applied fileset. When a fileset is committed on a server, there’s no way to roll back to a previous version of the software, short of a reinstall. The committed fileset is locked in place. When a fileset is applied, however, it can be rejected and removed from the server in the event that applying the fileset causes something to go wrong on the server.

When it’s time to apply some patches or upgrade a server, especially when dealing with TLs and SPs, I recommend starting with a mksysb backup of the server first. Then, after the backup is done, commit any applicable filesets to cement the state of the OS. Following this procedure will provide a solid foundation and restore point for the rest of the process.

Install the software in the applied state so that in the event that something is off-kilter with the patches or upgrade, it can be promptly rejected. If you can afford to do so (even if the software doesn’t require it), I advise that you also reboot the server afterward to make sure that the system periodically gets tested and any latent problems can be shaken out. After a few weeks of the server running well and going through a vetting period, commit the software to complete the process and set the new baseline for the server.

Stick with the Big Groupings

Sometimes an application will require an individual OS fileset to be upgraded in order to work properly. For example, the pre-check for installing Oracle will often call out several filesets to be patched before the database software can be installed. However, I never advise upgrading filesets on a one-off basis; I always stick with the practice of upgrading through TLs or SPs.

When admins deviate from this practice and place a number of incongruous updates on their servers, it makes administration more difficult in the long run. Commands like oslevel or instfix will return inconsistent numbers for the software levels that are running. Future patches may balk at having some components already at higher levels than they should be. Plus, if you have to contact IBM Support for help, it will make explaining your server much more complicated than simply saying, "I'm running AIX 6.1, TL 06, SP 05."

The only cases in which I would endorse deviating from this practice are if something is seriously broken on the server or if a high-profile vulnerability could impact the server. In such cases, it’s appropriate to add an IBM eFix and resume the recommended patching practice after IBM integrates the fix into a future TL or SP.

Use NIM or a Software Repository

One big headache that plagues AIX systems admins is working in an environment where servers are all across the board with their OSs. I’ve been in some shops that ran every version of AIX, from 5.1 to 7.1, with multiple TLs and SPs in each version and release. The variation of OS versions made keeping track of servers a nightmare: finding which systems needed to be patched, which ones had vulnerabilities, and picking and choosing which needed immediate patching, especially when IBM emailed warnings about high-impact alerts.

The Network Installation Manager (NIM) utility makes it a breeze to install the same level of OS onto multiple servers at once. Find a good level of the AIX OS that works for your servers, install NIM onto a small LPAR, and turn that system into a bastion host that serves as a software fix repository. By doing this, you’ll only need to be concerned with one or two iterations of the OS, and you’ll find it easier to mass-deploy a homogenous OS configuration.

Watch for RPMs Versus Filesets

With the advent of Linux affinity in the AIX OS about a decade ago, it became possible to compile and install Red Hat packages onto Power Systems. While most OS patches pertain to AIX filesets, there are a fair number of applications that work closely with the OS that use RPM packages. These applications include utilities like sudo, wget, and gcc, which all have a role in administration.

When you go to patch your servers, be sure to also check the RPM packages installed on your system to determine whether they should also be updated. Use both the rpm and lslpp commands to check all those other pieces of software.

Use nimadm_migrate and alt_disk_install Strategies

One of the best features of AIX that has come out in the past few years is a combination of two technologies: nimadm_migrate, which is the AIX System Management Interface Tool (SMIT) fastpath to migrating a client box to a new version of the OS, and alt_disk_install. These two commands allow admins to take the currently running OS, clone it onto a blank hdisk, and then even apply patches onto that drive. This way, the server stays completely up and there’s no risk to the currently running root volume group (rootvg) as the server is migrated to a new SP, TL, or even a completely new version and release of the OS. All it takes is a reboot to switch over to the fresh OS. And in the event something goes wrong, the server just needs to be rebooted onto the original disk to put things back the way they were.

Stay Current with Patching

Keeping your servers up to date with the AIX OS and patching them on a routine schedule can stave off unnecessary downtime, give your users more features, and periodically "shake the tree" to see if anything falls out. We'll continue our exploration of AIX management in the next article, when we look at a few techniques to make networking more understandable and manageable.

Courtesy Christian Pruett &  POWER IT Pro.