Introduction to PowerHA

There is a good Introduction to PowerHA article from  IBM DeveloperWorks if you are new to PowerHA(formerly HACMP).

http://www.ibm.com/developerworks/aix/library/au-powerhaintro/index.html?ca=drs-

HACMP: Cluster Commandline

1. PowerHA Version

There is no specific command to get the PowerHA version. However, the version of the cluster.es.server.rte fileset actually reflects the PowerHA version

# lslpp -Lqc cluster.es.server.rte | cut -d: -f3
6.1.0.3

If you don't trust the above method to determine the PowerHA version you could also ask the cluster manager:

# lssrc -ls clstrmgrES | egrep '^local node vrmf|^cluster fix level'
local node vrmf is 6103
cluster fix level is "3"

2. Program Paths

The path to the cluster commands are not included in the default path. It makes sense to extend the default path to include the cluster paths:

# export PATH=$PATH:/usr/es/sbin/cluster:/usr/es/sbin/cluster/utilities:/usr/es/sbin/cluster/sbin:/usr/es/sbin/cluster/cspoc

Dynamic Reconfiguration (CSPOC)

1. Extend a Volume Group

One or more PVs can be added to an existing volume group. Since it is not guaranteed that the hdisks are numbered the same way across all nodes you need to specify a reference node with the "-R" switch:

nodeA# /usr/es/sbin/cluster/sbin/cl_extendvg -cspoc -n'nodeA,nodeB' -R'nodeA' VolumeGroup hdiskA hdiskB hdisk...

2. Reduce a Volume Group

nodeA# /usr/es/sbin/cluster/sbin/cl_reducevg -cspoc -n'nodeA,nodeB' -R'nodeA' VolumeGroup hdiskA hdiskB hdisk...

Again a reference node has to be specified.

3. Add a Filesystem to an Existing VG

nodeA# /usr/es/sbin/cluster/sbin/cl_mklv -cspoc -n'nodeA,nodeB' -R'nodeA' -y'LVName' -t'jfs2' -e'x' -u'2' -s's' VolumeGroup LPs hdiskA hdiskB

It is recommended to use the most narrow upperbound possible to keep the mirror consistant after a filesystem extension.

You could also use a map file to tell the command how to setup your LV:

nodeA# /usr/es/sbin/cluster/sbin/cl_mklv -cspoc -n'nodeA,nodeB' -R'nodeA' -y'LVName' -t'jfs2' -m MapFile VolumeGroup LPs
nodeA# /usr/es/sbin/cluster/cspoc/cli_chlv -e x -u'2' -s's'  LVName

The format of the map file is

 hdiskA:PP1
 hdiskB:PP1
    :
 hdiskC:PP2
 hdiskD:PP2

First put in all mappings for mirror copy 1, then add all mapping for mirror copy 2. The number of entries per mirror copy has to equal the number of LPs given at the command line. Be careful!

Then we create the filessystem on top of the just created LV...

nodeA# /usr/es/sbin/cluster/sbin/cl_crfs -v'jfs2' -m'/mountpoint' -d'/dev/LVName' -A'n' -a'logname=INLINE'

4. Increase a Filesystem

If there are still enough free PPs in the existing Volume group the filesystem can be extended with a standard AIX command:

nodeA# chfs -a size=512G /mountpoint

Be sure that superstrictness is set as well as that the upper bound is set correctly.

In case you need to add new disks to the VG (1. Extend a Volume Group) in order to be able to extend the filesystem you need to extend the underlying LV first:

nodeA# /usr/es/sbin/cluster/sbin/cl_extendlv -u'8' -m'MapFile' LVName LPs
nodeA# chfs -a size=512G /mountpoint

The upperbound has to be adapted to the new number of PVs. The map file must only contain the additional mappings.

5. Mirror a Logical Volume

# /usr/es/sbin/cluster/sbin/cl_mklvcopy -R'NODE' -e'x' LVName 2 hdiskA hdiskB hdisk...

Since it is not guaranteed that the hdisks are numbered the same way across all nodes you need to specify a reference node.

6. Remove a Logical Volume Mirror

# /usr/es/sbin/cluster/sbin/cl_rmlvcopy -R'NODE' LVName 1 hdiskA hdiskB hdisk...

Again a reference node has to be specified.

Resource Group Management

1. Move a Resource Group to another Node

# /usr/es/sbin/cluster/utilities/clRGmove -g RG -n NODE -m

It is also possible to move multiple resource groups in one go:

# /usr/es/sbin/cluster/utilities/clRGmove -g "RG1,RG2,RG3" -n NODE -m

2. Bring a Resource Group Down

# /usr/es/sbin/cluster/utilities/clRGmove -g RG -n NODE -d

Multiple resource groups can be concatenated with commas: -g "RG1,RG2,RG3"

3. Bring a Resource Group Up

# /usr/es/sbin/cluster/utilities/clRGmove -g RG -n NODE -u

Multiple resource groups can be concatenated with commas: -g "RG1,RG2,RG3"

Cluster Information

1. Where Can I Find the Log Files?

Historically the cluster's main log file could be found under "/tmp/hacmp.out". But nowadays the location is configurable. If you don't know where to look run

# /usr/es/sbin/cluster/utilities/cllistlogs
/var/hacmp/log/hacmp.out
/var/hacmp/log/hacmp.out.1
/var/hacmp/log/hacmp.out.2

2. Where Can I Finde the Application Start/Stop Scripts?

# /usr/es/sbin/cluster/utilities/cllsserv
AppSrv1     /etc/cluster/start_appsrv1     /etc/cluster/stop_appsrv1
AppSrv2     /etc/cluster/start_appsrv2     /etc/cluster/stop_appsrv2

3. Show the Configuration of a Particular Resource Group

# /usr/es/sbin/cluster/utilities/clshowres -g RG

If you are interested in the configuration of all resource groups you can use clshowres without any option:

# /usr/es/sbin/cluster/utilities/clshowres

4. Cluster IP Configuration

# /usr/es/sbin/cluster/utilities/cllsif

Cluster State

1. Cluster State

The most widely known tool to check the cluster state is probably clstat:

# /usr/es/sbin/cluster/clstat -a

The switch "-a" forces clstat to run in terminal mode rather to than open an X window.

The same information can be obtained with

# /usr/es/sbin/cluster/utilities/cldump

clstat and cldump rely on snmp to work.

2. The Cluster Manager

If for whatever reason snmp does not allow you to use clstat or cldump you can still ask the cluster manager about the state of your cluster:

# lssrc -ls clstrmgrES
Current state: ST_STABLE
sccsid = "@(#)36    1.135.1.101 src/43haes/usr/sbin/cluster/hacmprd/main.C, hacmp.pe, 53haes_r610 11/16/10 06:18:14"
i_local_nodeid 0, i_local_siteid -1, my_handle 1
ml_idx[1]=0     ml_idx[2]=1    
There are 0 events on the Ibcast queue
There are 0 events on the RM Ibcast queue
CLversion: 11
local node vrmf is 6103
cluster fix level is "3"
The following timer(s) are currently active:
Current DNP values
DNP Values for NodeId - 1  NodeName - barney
    PgSpFree = 16743822  PvPctBusy = 3  PctTotalTimeIdle = 90.121875
DNP Values for NodeId - 2  NodeName - betty
    PgSpFree = 16746872  PvPctBusy = 0  PctTotalTimeIdle = 97.221894

The most important informations are marked red.

3. Where are the Resources Currently Active?

# /usr/es/sbin/cluster/utilities/clRGinfo
-----------------------------------------------------------------------------
Group Name     Group State                  Node           
-----------------------------------------------------------------------------
RES_GRP_01     ONLINE                       barney       
               OFFLINE                      betty       

RES_GRP_02     ONLINE                       betty       
               OFFLINE                      barney        

Useful HACMP commands

clstat - show cluster state and substate; needs clinfo. If clinfo is inoperative this command will not work.

cldump - SNMP-based tool to show cluster state

cldisp - similar to cldump, perl script to show cluster state.

cltopinfo - list the local view of the cluster topology.

clshowsrv -a - list the local view of the cluster subsystems.

clfindres (-s) - locate the resource groups and display status.

clRGinfo -v - locate the resource groups and display status.

clrsh - cluster rsh program that take cluster node names as argument.

clgetactivenodes - which nodes are active?

get_local_nodename - what is the name of the local node?

clconfig - check the HACMP ODM.

cllsgrp - list the resource groups.

cllscf - list the network configuration of an hacmp cluster.

clshowres - show the resource group configuration.

cllsif - show network interface information.

cllsres - show short resource group information.

lssrc -ls clstrmgrES - list the cluster manager state.

lssrc -ls topsvcs - show heartbeat information.

cllsnode - list a node centric overview of the hacmp configuration.


For any operations like movement of RG or making them online/offline etc...use smitty hacmp  (Recommended).

Commands are -

clcycle - rotate some of the log files.

clRGmove - online/offline or move resource groups.

cldare - sync/fix the cluster.

clsnapshotinfo - create a large snapshot of the hacmp configuration.