Solaris ODS (online disk suite)

Sun's volume manager has many names

• Online Disk Suite (ODS) - will be using this name in this document
• Solstice Disk Suite (SDS)
• Solaris Logical Volume Manager (Solaris LVM)

ODS is a disk storage management solution, which offers

• High Availability
• Improved Performance
• Simplified disk management

Raid Levels
The disk management software offers the common raid levels

raid 0 (Striping)	A number of disks are concatenated together to give the appearance of one very large disk. Advantages    Improved performance    Can Create very large Volumes Disadvantages    Not highly available (if one disk fails, the volume fails)
raid 1 (Mirroring)	A single disk is mirrored by another disk, if one disk fails the system is unaffected as it can use its mirror. Advantages    Improved performance    Highly Available (if one disk fails the mirror takes over) Disadvantages    Expensive (requires double the number of disks)
raid 5	Raid stands for Redundant Array of Inexpensive Disks, the disks are striped with parity across 3 or more disks, the parity is used in the event that one of the disks fails, the data on the failed disk is reconstructed by using the parity bit. Advantages    Improved performance (read only)    Not expensive Disadvantages    Slow write operations (caused by having to create the parity bit)

Metadevice and Metadevice Database
A metadevice is a name for a group of physical slices that appear as a single logical device (virtual device). The maximum default number of metadevices is 128 but this can be adjusted by editing /kernel/drv/md.conf and changing the nmd parameter (1024 maximum).
A metadevice database (otherwise know as state database) is a database that stores information about the ODS configuration, it is used to store and track changes made to ODS, this database is what makes the ODS persistent across reboots. The database has multiple copies known as replicas (minimum of 3 is required), this ensures that the database is always valid, you should keep multiple copies across different disks just in case a disk should fail and thus reducing single-points of failure, the database is never more than an 10MB and is generally stored on a single slice of each disk.
ODS uses a majority consensus algorithm to determine if a replica is corrupted or not, when changes are made each replica is updated in turn just in case a power failure happens during the update, thus when the system is started the majority replicas will be implemented, the algorithm guarantees the following

• The system will stay running with exactly half or more state database replicas
• The system will panic if more than half the state database replicas are not available
• The system will not reboot without one more than half the total state database replicas

Hot Spares
ODS uses a hot spare pool, which is a collection of disk slices reserved by ODS which will automatically be used when a disk slice fails. They provide increased data protection, however i have very rarely used hot spares as i normally replace a failed disk pretty quickly. See the Sun Documentation for detail information on hot spares.
Growing/Shrinking Filesystem
Expanding filesystems is not without problems with ODS but it is possible, however shrinking a filesystem under ODS is not possible, normally you create another new smaller filesystem and copy the data across then cut over to the new filesystem.
This is one area the Veritas volume manager excels as it very easy to grow and shrink a filesystem.
Filesystem Logging
ODS uses translogs to log changes made to the filesystem, in the event that the system were to crash the log is replayed thus avoiding a fsck (which can take a long time depending on the size of the filesystem). However newer versions of Solaris offer UFS logging, here is a list of advantages/disadvantages of both
ODS logging

• Can be mirrored and therefore survive better from disk failures
• Does not support root filesystem

UFS logging

• Simple to implement (just update /etc/vfstab and add logging option)
• Does not require it's own slice
• Supports root filessytem
• Tighter connection to the unix kernel which results in less overhead

My preference is to use UFS logging and since its introduction in solaris 7 i have only ever used this.
Naming Convention
There is no set standard on what you call your metadevices but i have my own convention and undoubtedly there are many others.
The main metadevice (raid 0,1 or 5) which is were the filesystem will be placed will always end in 0 so for example d0, d10, d20, d30, d40, etc
A sub-mirror will either end in a 1 (first sub-mirror) or 2 (second sub-mirror) so for example d1 and d2, d11 and d12, d21 and d22, etc
A raid slice will either end in a 1..n (n = depends on number of disks) so for example d1 & d2 & d3, d21 & d22 & d23, etc
So for an example

• Mirrored metadevice - I would create the mirror metadevice as d0 and have to sub-mirrors called d1, d2
• Mirrored metadevice - I would create the mirror metadevice as d10 and have to sub-mirrors called d11, d12
• Raid 5 device - I would create the raid device as d20 and have the raid-slices called d21, d22, d23

This is my own preference and you are welcome to have your own naming convention
File Locations
ODS uses a number of different files, below are the most useful one's:

/kernel/drv/md.conf	This file is the ODS device drive configuration file, the only modifiable field is the 'nmd' which represents the number of metadevices supported by the driver, if you change this file you must reboot the system for the changes to take affect. In a configuration that uses a lot of devices I increase this to the maximum 1024.
/etc/lvm/mddb.cf	This file keeps track of metadevice state database replicas, each metadevice state database has a unique entry in this file. You can display the file using 'cat' but do not edit it manually.
/etc/lvm/md.tab	This file is used by metainit, metadb and metahs commands. The file contains the the rest of the commandline for use by metainit, metadb and metahs. This file can be edited manually or populated by the command 'metastat -p'
/etc/lvm/md.cf	This file is a copy of the md.tab file and is used for disaster recovery purposes, it is automatically updated.

Meta Commands
I am not going to explain in details how ODS works but simply supply a list of commands that I use regularly, if you want a more detail explanation then I suggest you refer to the Sun Documentation

Metadatabase Commands
Create	metadb -a -f -c 3 c0t0d0s6 c1t0d0s6 c2t0d0s6 -a - attach metadatabase to device -f - create the initial metadatabase and force deletion of replicas below the minimum of one -c - specifies the number of replicas to be placed on each device
Add	metadb -a -c 3 c3t0d0s6
Remove	metadb -d c3t0d0s6
Display	metadb -i
Repairing	# The only way to repair a replica is that you simply delete all the replica's on the device and # recreate them # First confirm that the replicas are corrupted and you have the device name metadb -i # Delete the corrupted replicas and reboot metadb -d c3t0d0s6 reboot # Now recreate them making sure you have 3 copies metadb -a -c 3 c3t0d0s6 metadb -i
Metadevice Commands
Create Concatenated device	metainit d0 3 1 c1t0d0s0 1 c2t0d0s0 1 c3t0d0s0 d0 - metadevice name 3 - total number of slices 1 c1t0d0s0 - number of slices to added followed by device name
Create stripe metadevice	metainit d0 1 2 c1t0d0s0 c2t0d0s0 -i 64k d0 - metadevice name 1 - total number of stripes 2 c1t0d0s0 c2t0d0s0 - number of slices to be added to stripe followed by device name i 64k - stripe size
Create Mirror metadevice	# first create two metadevices (these will become sub-mirrors) metainit d11 1 1 c2t0d0s0 metainit d12 1 1 c3t0d0s0 # Then create the mirror metadevice using the metadevice d11 (now called a sub-mirror) metainit d10 -m d11 # Then attach the second sub-mirror using the metadevice d12 create above to the mirror d10 metattach d10 d12 # Display the mirrored metadevice and confirm that mirror has complete resyncing operation # this may take a long time depending on the size of the mirror device metastat d10
Create Raid 5 metadevice	# When creating a raid 5 metadevice you need a minimum of 3 slices metainit d10 -r c1t0d0s0 c2t0d0s0 c3t0d0s0 -r - specify that its a raid 5 configuration
Mirroring the root filesystem	# Lets say you want to mirror the main disk which has the following filesystems configured, we will be using # c1t0d0 as the new mirror disk # # We hope to achieve the following device configuration # d0 - mirrored metadevice which contains the root filesystem #    d1 - a sub-mirror metadevice of d0 (c0t0d0s0) #    d2 - a sub-mirror metadevice of d0 (c1t0d0s0) # # If either c0t0d0s0 or c1t0d0s0 fails the other will take over, thus the system will continue to work as normal # The first step is to make sure the partition information is the same on the new mirror disk (c1t0d0) # basically copies the partition information to the new mirror device prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c1t0d0s2 # Then we want to install the boot block on the new mirror device, this allows you boot the disk should # the other disk fails installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t0d0s0 # Create first metadevice which will become the a sub-mirror of d0 # NOTE: although we are using the existing root slice this does not delete any data, we have also # specified the -f (force) option as the filesystem is mounted metainit -f d1 1 1 c0t0d0s0 # Create the second metadevice which will become the sub-mirror of d0, we do not need the -f option (force) # as there is not filesystem on the new device metainit d2 1 1 c1t0d0s0 # At this point we have two metadevices d11 (contains root filesystem) and d12 (the new disk) # we now create the mirror metadevice d0 metainit -d0 -m d1 # We now have to update the /etc/system and /etc/vfstab with the new root metadevice information metaroot d0 # Now reboot the server so that the new mirror metadevice is mounted and the kernel parameters for ODS # are loaded, we lock the filesystem before rebooting making sure all buffers have been written to the # filesystem lockfs -fa reboot # Once the server has been rebooted attach the second sub-mirror metattach d0 d12 # Depending on how big the root filesystem the longer the resyncing of the two mirrors will take metastat d0 # Once the mirrors are sync'ed you have a root filesystem that is highly available, you can now perform # the same task with other filesystems such as /var, swap, /usr, etc
Other ODS Commands
Display Metadatabse	metadb -i
Display Metadevices	metastat
Display metadevice in md.tab format	metastat -p

ODS Errors
A list of some of the more common errors of ODS

"no such file or directory error" when trying to configure a metadevice

# update the nmd parameter in the /kernel/drv/md.conf file, i normally increase this to it's maximum 1024.

Metadevice in maintenance state

# Disks do go bad from time to time, however there is a difference between a total disk failure or a # disk with bad data blocks, however if you replace the disk and use the same disk slice then the same # command is used # First access the disk via format, if you can then run a analyze on the disk to repair/map out any bad # data blocks format -> select disk -> anal -> read # If you cannot access the disk via format then physically replace the disk, then run the below command # to repair ODS, you must do this for each metadevice configured for that disk metareplace -e d0 c1t0d0s0 # If you want to replace the disk with a different disk then run metareplace d0 c1t0d0s0 <new device name> # Again confirm that the disk re-sync'ed metastat d0

AWK Cheat Sheet

 ===================== Predefined Variable Summary =====================

.-------------+-----------------------------------.---------------------.
|             |                                   |      Support:       |
| Variable    | Description                       '-----.-------.-------'
|             |                                   | AWK | NAWK  | GAWK  |
'-------------+-----------------------------------+-----+-------+-------'
| FS          | Input Field Separator, a space by |  +  |   +   |   +   |
|             | default.                          |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| OFS         | Output Field Separator, a space   |  +  |   +   |   +   |
|             | by default.                       |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| NF          | The Number of Fields in the       |  +  |   +   |   +   |
|             | current input record.             |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| NR          | The total Number of input Records |  +  |   +   |   +   |
|             | seen so far.                      |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| RS          | Record Separator, a newline by    |  +  |   +   |   +   |
|             | default.                          |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| ORS         | Output Record Separator, a        |  +  |   +   |   +   |
|             | newline by default.               |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| FILENAME    | The name of the current input     |     |       |       |
|             | file. If no files are specified   |     |       |       |
|             | on the command line, the value of |     |       |       |
|             | FILENAME is "-". However,         |  +  |   +   |   +   |
|             | FILENAME is undefined inside the  |     |       |       |
|             | BEGIN block (unless set by        |     |       |       |
|             | getline).                         |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| ARGC        | The number of command line        |     |       |       |
|             | arguments (does not include       |     |       |       |
|             | options to gawk, or the program   |  -  |   +   |   +   |
|             | source). Dynamically changing the |     |       |       |
|             | contents of ARGV control the      |  -  |   +   |   +   |
|             | files used for data.              |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| ARGV        | Array of command line arguments.  |     |       |       |
|             | The array is indexed from 0 to    |  -  |   +   |   +   |
|             | ARGC - 1.                         |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| ARGIND      | The index in ARGV of the current  |  -  |   -   |   +   |
|             | file being processed.             |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| BINMODE     | On non-POSIX systems, specifies   |     |       |       |
|             | use of "binary" mode for all file |     |       |       |
|             | I/O.Numeric values of 1, 2, or 3, |     |       |       |
|             | specify that input files, output  |     |       |       |
|             | files, or all files, respectively,|     |       |       |
|             | should use binary I/O. String     |     |       |       |
|             | values of  "r", or "w" specify    |  -  |   -   |   +   |
|             | that input files, or output files,|     |       |       |
|             | respectively, should use binary   |     |       |       |
|             | I/O. String values of "rw" or     |     |       |       |
|             |  "wr" specify that all files      |     |       |       |
|             | should use binary I/O. Any other  |     |       |       |
|             | string value is treated as "rw",  |     |       |       |
|             | but generates a warning message.  |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| CONVFMT     | The CONVFMT variable is used to   |     |       |       |
|             | specify the format when           |  -  |   -   |   +   |
|             | converting a number to a string.  |     |       |       |
|             | Default: "%.6g"                   |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| ENVIRON     | An array containing the values    |  -  |   -   |   +   |
|             | of the current environment.       |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| ERRNO       | If a system error occurs either   |     |       |       |
|             | doing a redirection for getline,  |     |       |       |
|             | during a read for getline, or     |     |       |       |
|             | during a close(), then ERRNO will |  -  |   -   |   +   |
|             | contain a string describing the   |     |       |       |
|             | error. The value is subject to    |     |       |       |
|             | translation in non-English locales.     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| FIELDWIDTHS | A white-space separated list of   |     |       |       |
|             | fieldwidths. When set, gawk       |     |       |       |
|             | parses the input into fields of   |  -  |   -   |   +   |
|             | fixed width, instead of using the |     |       |       |
|             | value of the FS variable as the   |     |       |       |
|             | field separator.                  |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| FNR         | Contains number of lines read,    |  -  |   +   |   +   |
|             | but is reset for each file read.  |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| IGNORECASE  | Controls the case-sensitivity of  |     |       |       |
|             | all regular expression and string |     |       |       |
|             | operations. If IGNORECASE has a   |     |       |       |
|             | non-zero value, then string       |     |       |       |
|             | comparisons and pattern matching  |     |       |       |
|             | in rules, field splitting         |     |       |       |
|             | with FS, record separating        |     |       |       |
|             | with RS, regular expression       |     |       |       |
|             | matching with ~ and !~, and the   |  -  |   -   |   +   |
|             | gensub(), gsub(), index(),        |     |       |       |
|             | match(), split(), and sub()       |     |       |       |
|             | built-in functions all ignore     |     |       |       |
|             | case when doing regular           |     |       |       |
|             | expression operations.            |     |       |       |
|             | NOTE: Array subscripting is not   |     |       |       |
|             | affected. However, the asort()    |     |       |       |
|             | and asorti() functions are        |     |       |       |
|             | affected                          |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| LINT        | Provides dynamic control of the   |     |       |       |
|             | --lint option from within an AWK  |  -  |   -   |   +   |
|             | program. When true, gawk prints   |     |       |       |
|             | lint warnings.                    |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| OFMT        | The default output format for     |  -  |   +   |   +   |
|             | numbers. Default: "%.6g"          |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| PROCINFO    | The elements of this array        |     |       |       |
|             | provide access to information     |     |       |       |
|             | about the running AWK program.    |     |       |       |
|             | PROCINFO["egid"]:                 |     |       |       |
|             | the value of the getegid(2)       |     |       |       |
|             | system call.                      |     |       |       |
|             | PROCINFO["euid"]:                 |     |       |       |
|             | the value of the geteuid(2)       |     |       |       |
|             | system call.                      |     |       |       |
|             | PROCINFO["FS"]:                   |     |       |       |
|             | "FS" if field splitting with FS   |     |       |       |
|             | is in effect, or "FIELDWIDTHS"    |     |       |       |
|             | if field splitting with           |     |       |       |
|             | FIELDWIDTHS is in effect.         |     |       |       |
|             | PROCINFO["gid"]:                  |  -  |   -   |   +   |
|             | the value of the getgid(2) system |     |       |       |
|             | call.                             |     |       |       |
|             | PROCINFO["pgrpid"]:               |     |       |       |
|             | the process group ID of the       |     |       |       |
|             | current process.                  |     |       |       |
|             | PROCINFO["pid"]:                  |     |       |       |
|             | the process ID of the current     |     |       |       |
|             | process.                          |     |       |       |
|             | PROCINFO["ppid"]:                 |     |       |       |
|             | the parent process ID of the      |     |       |       |
|             | current process.                  |     |       |       |
|             | PROCINFO["uid"]                   |     |       |       |
|             | the value of the getuid(2) system |     |       |       |
|             | call.                             |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| RT          | The record terminator. Gawk sets  |     |       |       |
|             | RT to the input text that matched |  -  |   -   |   +   |
|             | the character or regular          |     |       |       |
|             | expression specified by RS.       |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| RSTART      | The index of the first character  |  -  |   +   |   +   |
|             | matched by match(); 0 if no match.|     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| RLENGTH     | The length of the string matched  |  -  |   +   |   +   |
|             | by match(); -1 if no match.       |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| SUBSEP      | The character used to separate    |     |       |       |
|             | multiple subscripts in array      |     |       |       |
|             | elements.Default: "\034"          |  -  |   +   |   +   |
|             | (non-printable character,         |     |       |       |
|             | dec: 28, hex: 1C)                 |     |       |       |
'-------------+-----------------------------------+-----+-------+-------'
| TEXTDOMAIN  | The text domain of the AWK        |     |       |       |
|             | program; used to find the         |  -  |   -   |   +   |
|             | localized translations for the    |     |       |       |
|             | program's strings.                |     |       |       |
'-------------'-----------------------------------'-----'-------'-------'


 ============================ I/O Statements ===========================

.---------------------.-------------------------------------------------.
|                     |                                                 |
| Statement           | Description                                     |
|                     |                                                 |
'---------------------+-------------------------------------------------'
| close(file [, how]) | Close file, pipe or co-process. The optional    |
|                     | how should only be used when closing one end of |
|                     | a two-way pipe to a co-process. It must be a    |
|                     | string value, either "to" or "from".            |
'---------------------+-------------------------------------------------'
| getline             | Set $0 from next input record; set NF, NR, FNR. |
|                     | Returns 0 on EOF and �1 on an error. Upon an    |
|                     | error, ERRNO contains a string describing the   |
|                     | problem.                                        |
'---------------------+-------------------------------------------------'
| getline <file       | Set $0 from next record of file; set NF.        |
'---------------------+-------------------------------------------------'
| getline var         | Set var from next input record; set NR, FNR.    |
'---------------------+-------------------------------------------------'
| getline var <file   | Set var from next record of file.               |
'---------------------+-------------------------------------------------'
| command |           | Run command piping the output either into $0 or |
|   getline [var]     | var, as above. If using a pipe or co-process    |
|                     | to getline, or from print or printf within a    |
|                     | loop, you must use close() to create new        |
|                     | instances                                       |
'---------------------+-------------------------------------------------'
| command |&          | Run command as a co-process piping the output   |
|   getline [var]     | either into $0 or var, as above.  Co-processes  |
|                     | are a gawk extension.                           |
'---------------------+-------------------------------------------------'
| next                | Stop processing the current input record.       |
|                     | The next input record is read and processing    |
|                     | starts over with the first pattern in the AWK   |
|                     | program. If the end of the input data is        |
|                     | reached, the END block(s), if any, are executed.|
'---------------------+-------------------------------------------------'
| nextfile            | Stop processing the current input file. The     |
|                     | next input record read comes from the next      |
|                     | input file. FILENAME and ARGIND are updated,    |
|                     | FNR is reset to 1, and processing starts over   |
|                     | with the first pattern in the AWK program. If   |
|                     | the end of the input data is reached, the END   |
|                     | block(s), are executed.                         |
'---------------------+-------------------------------------------------'
| print               | Prints the current record. The output record is |
|                     | terminated with the value of the ORS variable.  |
'---------------------+-------------------------------------------------'
| print expr-list     | Prints expressions. Each expression is          |
|                     | separated by the value of the OFS variable.     |
|                     | The output record is terminated with the value  |
|                     | of the ORS variable.                            |
'---------------------+-------------------------------------------------'
| print expr-list     | Prints expressions on file. Each expression is  |
|   >file             | separated by the value of the OFS variable. The |
|                     | output record is terminated with the value of   |
|                     | the ORS variable.                               |
'---------------------+-------------------------------------------------'
| printf fmt,         | Format and print.                               |
|   expr-list         |                                                 |
'---------------------+-------------------------------------------------'
| printf fmt,         | Format and print on file.                       |
|   expr-list >file   |                                                 |
'---------------------+-------------------------------------------------'
| system(cmd-line)    | Execute the command cmd-line, and return the    |
|                     | exit status.                                    |
'---------------------+-------------------------------------------------'
| fflush([file])      | Flush any buffers associated with the open      |
|                     | output file or pipe file. If file is missing,   |
|                     | then stdout is flushed. If file is the null     |
|                     | string, then all open output files and pipes    |
|                     | have their buffers flushed.                     |
'---------------------+-------------------------------------------------'
| print ... >> file   | Appends output to the file.                     |
'---------------------+-------------------------------------------------'
| print ... | command | Writes on a pipe.                               |
'---------------------+-------------------------------------------------'
| print ... |&        | Sends data to a co-process.                     |
|   command           |                                                 |
'---------------------'-------------------------------------------------'


 =========================== Numeric Functions =========================

.---------------------.-------------------------------------------------.
|                     |                                                 |
| Function            | Description                                     |
|                     |                                                 |
'---------------------+-------------------------------------------------'
| atan2(y, x)         | Returns the arctangent of y/x in radians.       |
'---------------------+-------------------------------------------------'
| cos(expr)           | Returns the cosine of expr, which is in radians.|
'---------------------+-------------------------------------------------'
| exp(expr)           | The exponential function.                       |
'---------------------+-------------------------------------------------'
| int(expr)           | Truncates to integer.                           |
'---------------------+-------------------------------------------------'
| log(expr)           | The natural logarithm function.                 |
'---------------------+-------------------------------------------------'
| rand()              | Returns a random number N, between 0 and 1,     |
|                     | such that 0 <= N < 1.                           |
'---------------------+-------------------------------------------------'
| sin(expr)           | Returns the sine of expr, which is in radians.  |
'---------------------+-------------------------------------------------'
| sqrt(expr)          | The square root function.                       |
'---------------------+-------------------------------------------------'
| srand([expr])       | Uses expr as a new seed for the random number   |
|                     | generator. If no expr is provided, the time of  |
|                     | day is used. The return value is the previous   |
|                     | seed for the random number generator.           |
'---------------------'-------------------------------------------------'


 ====================== Bit Manipulation Functions =====================

.---------------------.-------------------------------------------------.
|                     |                                                 |
| Function            | Description                                     |
|                     |                                                 |
'---------------------+-------------------------------------------------'
| and(v1, v2)         | Return the bitwise AND of the values provided   |
|                     | by v1 and v2.                                   |
'---------------------+-------------------------------------------------'
| compl(val)          | Return the bitwise complement of val.           |
'---------------------+-------------------------------------------------'
| lshift(val, count)  | Return the value of val, shifted left by        |
|                     | count bits.                                     |
'---------------------+-------------------------------------------------'
| or(v1, v2)          | Return the bitwise OR of the values provided by |
|                     | v1 and v2.                                      |
'---------------------+-------------------------------------------------'
| rshift(val, count)  | Return the value of val, shifted right by       |
|                     | count bits.                                     |
'---------------------+-------------------------------------------------'
| xor(v1, v2)         | Return the bitwise XOR of the values provided   |
|                     | by v1 and v2.                                   |
'---------------------'-------------------------------------------------'


 =========================== String Functions ==========================

.---------------------.-------------------------------------------------.
|                     |                                                 |
| Function            | Description                                     |
|                     |                                                 |
'---------------------+-------------------------------------------------'
| asort(s [, d])      | Returns the number of elements in the source    |
|                     | array s.  The contents of s are sorted using    |
|                     | gawk's normal rules for comparing values, and   |
|                     | the indexes of the sorted values of s are       |
|                     | replaced with sequential integers starting with |
|                     | 1. If the optional destination array d is       |
|                     | specified, then s is first duplicated into d,   |
|                     | and then d is sorted, leaving the indexes of    |
|                     | the source array s unchanged.                   |
'---------------------+-------------------------------------------------'
| asorti(s [, d])     | Returns the number of elements in the source    |
|                     | array s. The behavior is the same as that of    |
|                     | asort(),  except that the array indices are     |
|                     | used for sorting, not the array values. When    |
|                     | done, the array is indexed numerically, and the |
|                     | values are those of the original indices. The   |
|                     | original values are lost; thus provide a second |
|                     | array if you wish to preserve the original.     |
'---------------------+-------------------------------------------------'
| gensub(r, s,        | Search the target string t for matches of the   |
|   h [, t])          | regular expression r.  If h is a string         |
|                     | beginning with g or G, then replace all matches |
|                     | of r with s. Otherwise, h is a number           |
|                     | indicating which match of r to replace. If t is |
|                     | not supplied, $0 is used instead. Within the    |
|                     | replacement text s, the sequence \n, where n is |
|                     | a digit from 1 to 9, may be used to indicate    |
|                     | just the text that matched the n'th             |
|                     | parenthesized subexpression. The sequence \0    |
|                     | represents the entire matched text, as does the |
|                     | character &. Unlike sub() and gsub(), the       |
|                     | modified string is returned as the result of    |
|                     | the function, and the original target string    |
|                     | is not changed.                                 |
'---------------------+-------------------------------------------------'
| gsub(r, s [, t])    | For each substring matching the regular         |
|                     | expression r in the string t, substitute the    |
|                     | string s, and return the number of              |
|                     | substitutions.  If t is not supplied, use $0.   |
|                     | An & in the replacement text is replaced with   |
|                     | the text that was actually matched. Use \& to   |
|                     | get a literal &.  (This must be                 |
|                     | typed as  "\\&")                                |
'---------------------+-------------------------------------------------'
| index(s, t)         | Returns the index of the string t in the        |
|                     | string s, or 0 if t is not present. (This       |
|                     | implies that characterindices start at one.)    |
'---------------------+-------------------------------------------------'
| length([s])         | Returns the length of the string s, or the      |
|                     | length of $0 if s is not supplied.              |
'---------------------+-------------------------------------------------'
| match(s, r [, a])   | Returns the position in s where the regular     |
|                     | expression r occurs, or 0 if r is not present,  |
|                     | and sets the values of RSTART and RLENGTH.      |
|                     | Note that the argument order is the same as for |
|                     | the ~ operator:  str  ~  re. If array a is      |
|                     | provided, a is cleared and then elements 1      |
|                     | through n are filled with the portions of s     |
|                     | that match the corresponding parenthesized      |
|                     | subexpression in r.  The 0'th element of a      |
|                     | contains the portion of s matched by the entire |
|                     | regular expression r. Subscripts a[n, "start"], |
|                     | and a[n, "length"] provide the starting index   |
|                     | in the string and length respectively, of each  |
|                     | matching substring.                             |
'---------------------+-------------------------------------------------'
| split(s, a [, r])   | Splits the string s into the array a on the     |
|                     | regular expression r, and returns the number of |
|                     | fields. If r is omitted, FS is used instead.    |
|                     | The array a is cleared first. Splitting behaves |
|                     | identically to field splitting.                 |
'---------------------+-------------------------------------------------'
| sprintf(fmt,        | Prints expr-list according to fmt, and returns  |
|   expr-list)        | the resulting string.                           |
'---------------------+-------------------------------------------------'
| strtonum(str)       | Examines str, and returns its numeric value.    |
|                     | If str begins with a leading 0, strtonum()      |
|                     | assumes that  str is an octal number. If str    |
|                     | begins with a leading 0x or 0X, strtonum()      |
|                     | assumes that str is a hexadecimal number.       |
'---------------------+-------------------------------------------------'
| sub(r, s [, t])     | Just like gsub(), but only the first matching   |
|                     | substring is replaced.                          |
'---------------------+-------------------------------------------------'
| substr(s, i [, n])  | Returns the at most n-character substring of s  |
|                     | starting at i.  If n is omitted, the rest of s  |
|                     | is used.                                        |
'---------------------+-------------------------------------------------'
| tolower(str)        | Returns a copy of the string str, with all the  |
|                     | upper-case characters in str translated to      |
|                     | their corresponding lower-case counterparts.    |
|                     | Non-alphabetic characters are left unchanged.   |
'---------------------+-------------------------------------------------'
| toupper(str)        | Returns a copy of the string str, with all the  |
|                     | lower-case characters in str translated to      |
|                     | their corresponding upper-case counterparts.    |
|                     | Non-alphabetic characters are left unchanged.   |
'---------------------'-------------------------------------------------'


 ============================ Time Functions ===========================

.---------------------.-------------------------------------------------.
|                     |                                                 |
| Function            | Description                                     |
|                     |                                                 |
'---------------------+-------------------------------------------------'
| mktime(datespec)    | Turns datespec into a time stamp of the same    |
|                     | form as returned by systime(). The datespec is  |
|                     | a string of the form YYYY MM DD HH MM SS[ DST]. |
|                     | The contents of the string are six or seven     |
|                     | numbers representing respectively the full year |
|                     | including century, the month from 1 to 12, the  |
|                     | day of the month from 1 to 31, the hour of the  |
|                     | day from 0 to 23, the minute from 0 to 59, and  |
|                     | the second from 0 to 60, and an optional        |
|                     | daylight saving flag. The values of these       |
|                     | numbers need not be within the ranges           |
|                     | specified; for example, an hour of -1 means 1   |
|                     | hour before midnight. The origin-zero Gregorian |
|                     | calendar is assumed, with year 0 preceding year |
|                     | 1 and year -1 preceding year 0. The time is     |
|                     | assumed to be in the local timezone. If the     |
|                     | daylight saving flag is positive, the time is   |
|                     | assumed to be daylight saving time; if zero,    |
|                     | the time is assumed to be standard time; and if |
|                     | negative (the default), mktime() attempts to    |
|                     | determine whether daylight saving time is in    |
|                     | effect for the specified time. If datespec does |
|                     | not contain enough elements or if the resulting |
|                     | time is out of range, mktime() returns -1.      |
'---------------------+-------------------------------------------------'
| strftime([format    | Formats timestamp according to the              |
|   [, timestamp]])   | specification in format. The timestamp should   |
|                     | be of the same form as returned by systime().   |
|                     | If timestamp is missing, the current time of    |
|                     | day is used.If format is missing, a default     |
|                     | format equivalent to the output of date(1) is   |
|                     | used.  See the specification for the strftime() |
|                     | function in ANSI C for the format conversions   |
|                     | that are guaranteed to be available. A          |
|                     | public-domain version of strftime(3) and a man  |
|                     | page for it come with gawk; if that version was |
|                     | used to build gawk, then all of the conversions |
|                     | described in that man page are available to     |
|                     | gawk.                                           |
'---------------------+-------------------------------------------------'
| systime()           | Returns the current time of day as the number   |
|                     | of seconds since the Epoch (1970-01-01 00:00:00 |
|                     | UTC on POSIX systems).                          |
'---------------------'-------------------------------------------------'


 =============== Internationalization (I18N)  Functions ================

.---------------------.-------------------------------------------------.
|                     |                                                 |
| Function            |                                                 |
|                     |                                                 |
| Description         |                                                 |
|                     |                                                 |
'---------------------+-------------------------------------------------'
| bindtextdomain(directory [, domain])                                  |
|                                                                       |
| Specifies the directory where gawk looks for the .mo files. It        |
| returns the directory where domain is ``bound.'' The default domain   |
| is the value of TEXTDOMAIN.  If directory is the null string (""),    |
| then bindtextdomain() returns the current binding for the given domain|
'---------------------+-------------------------------------------------'
| dcgettext(string [, domain [, category]])                             |
|                                                                       |
| Returns the translation of string in text domain domain for locale    |
| category category. The default value for domain is the current value  |
| of TEXTDOMAIN. The default value for category is "LC_MESSAGES". If    |
| you supply a value for category, it must be a string equal to one of  |
| the known locale categories. You must also supply a text domain. Use  |
| TEXTDOMAIN if you want to use the current domain.                     |
'---------------------+-------------------------------------------------'
| dcngettext(string1 , string2 , number [, domain [, category]])        |
|                                                                       |
| Returns the plural form used for number of the translation of string1 |
| and string2 in text domain domain for locale category category. The   |
| default value for domain is the current value of TEXTDOMAIN. The      |
| default value for category is "LC_MESSAGES". If you supply a value    |
| for category, it must be a string equal to one of the known locale    |
| categories. You must also supply a text domain. Use TEXTDOMAIN if     |
| you want to use the current domain.                                   |
'---------------------'-------------------------------------------------'




 =============== GNU AWK's Command Line Argument Summary ===============

.-------------------------.---------------------------------------------.
|                         |                                             |
| Argument                | Description                                 |
|                         |                                             |
'-------------------------+---------------------------------------------'
| -F fs                   | Use fs for the input field separator        |
| --field-sepearator fs   | (the value of the FS predefined variable).  |
'-------------------------+---------------------------------------------'
| -v var=val              | Assign the value val to the variable var,   |
| --assign var=val        | before execution of the program begins.     |
|                         | Such variable values are available to the   |
|                         | BEGIN block of an AWK program.              |
'-------------------------+---------------------------------------------'
| -f program-file         | Read the AWK program source from the file   |
| --file program-file     | program-file, instead of from the first     |
|                         | command line argument. Multiple -f          |
|                         | (or --file) options may be used.            |
'-------------------------+---------------------------------------------'
| -mf NNN                 | Set various memory limits to the value NNN. |
| -mr NNN                 | The f flag sets the maximum number of       |
|                         | fields, and the r flag sets the maximum     |
|                         | record size. (Ignored by gawk, since gawk   |
|                         | has no pre-defined limits)                  |
'-------------------------+---------------------------------------------'
| -W compat               | Run in compatibility mode. In compatibility |
| -W traditional          | mode, gawk behaves identically to UNIX awk; |
| --compat--traditional   | none of the GNU-specific extensions are     |
|                         | recognized.                                 |
'-------------------------+---------------------------------------------'
| -W copyleft             | Print the short version of the GNU copyright|
| -W copyright            | information message on the standard output  |
| --copyleft              | and exit successfully.                      |
| --copyright             |                                             |
'-------------------------+---------------------------------------------'
| -W dump-variables[=file]| Print a sorted list of global variables,    |
| --dump-variables[=file] | their types and final values to file. If no |
|                         | file is provided, gawk uses a file named    |
|                         | awkvars.out in the current directory.       |
'-------------------------+---------------------------------------------'
| -W help                 | Print a relatively short summary of the     |
| -W usage                | available options on the standard output.   |
| --help                  |                                             |
| --usage                 |                                             |
'-------------------------+---------------------------------------------'
|-W lint[=value]          | Provide warnings about constructs that      |
|--lint[=value]           | are dubious or non-portable to other AWK    |
|                         | impl�s. With argument fatal, lint warnings  |
|                         | become fatal errors. With an optional       |
|                         | argument of invalid, only warnings about    |
|                         | things that are actually invalid are        |
|                         | issued. (This is not fully implemented yet.)|
'-------------------------+---------------------------------------------'
| -W lint-old--lint-old   | Provide warnings about constructs that are  |
|                         | not portable to the original version of     |
|                         | Unix awk.                                   |
'-------------------------+---------------------------------------------'
| -W gen-po--gen-po       | Scan and parse the AWK program, and         |
|                         | generate a GNU .po format file on standard  |
|                         | output with entries for all localizable     |
|                         | strings in the program. The program itself  |
|                         | is not executed.                            |
'-------------------------+---------------------------------------------'
| -W non-decimal-data     | Recognize octal and hexadecimal values in   |
| --non-decimal-data      | input data.                                 |
'-------------------------+---------------------------------------------'
| -W posix--posix         | This turns on compatibility mode, with the  |
|                         | following additional restrictions:          |
|                         | o  \x escape sequences are not recognized.  |
|                         | o  Only space and tab act as field          |
|                         |    separators when FS is set to a single    |
|                         |    space, new-line does not.                |
|                         | o  You cannot continue lines after ? and :. |
|                         | o  The synonym func for the keyword function|
|                         |    is not recognized.                       |
|                         | o  The operators ** and **= cannot be used  |
|                         |    in place of ^ and ^=.�  The fflush()     |
|                         |    function is not available.               |
'-------------------------+---------------------------------------------'
| -W profile[=prof_file]  | Send profiling data to prof_file.           |
| --profile[=prof_file]   | The default is awkprof.out. When run with   |
|                         | gawk, the profile is just a "pretty         |
|                         | printed" version of the program. When run   |
|                         | with pgawk, the profile contains execution  |
|                         | counts of each statement in the program     |
|                         | in the left margin and function call counts |
|                         | for each user-defined function.             |
'-------------------------+---------------------------------------------'
| -W re-interval          | Enable the use of interval expressions in   |
| --re-interval           | regular expression matching. Interval       |
|                         | expressions were not traditionally          |
|                         | available in the AWK language.              |
'-------------------------+---------------------------------------------'
| -W source program-text  | Use program-text as AWK program source      |
| --source program-text   | code. This option allows the easy           |
|                         | intermixing of library functions (used via  |
|                         | the -f and --file options) with source code |
|                         | entered on the command line.                |
'-------------------------+---------------------------------------------'
| -W version              | Print version information for this          |
| --version               | particular copy of gawk on the standard     |
|                         | output.                                     |
'-------------------------+---------------------------------------------'
| --                      | Signal the end of options. This is useful   |
|                         | to allow further arguments to the AWK       |
|                         | program itself to start with a "-". This    |
|                         | is mainly for consistency with the argument |
|                         | parsing convention used by most other POSIX |
|                         | programs.                                   |
'-------------------------'---------------------------------------------'

 =======================================================================

AIX LVM Cheat Sheet

This is a quick and dirty cheat sheet on LVM using AIX, I have highlighted many of the common attributes for each command however this is not an extensive list, make sure you look up the command.

First a quick review on some of the terminology that AIX LVM uses

Examples	What it means
PHYSICAL VOLUME (PV)	Represents a hard disk (hdisk0).
PHYSICAL PARTITION (PP)	The smallest allocation unit in the LVM. All PPs within a VG are the same size, usually 4 or 8 MB.
VOLUME GROUP (VG)	A set of one or more PVs which form a single storage pool. You can define multiple VGs on each AIX system.
LOGICAL VOLUME (LV)	One or more PPs. A file system resides on top of an LV. Only one LV is mapped to a file system. A LV can't span across a VG. Up to 255 LVs in a VG
LOGICAL PARITITION (LP)	One or more PPs. LP represents a mirrored copy of a PP. Up to two copies of a PP can be mirrored resulting in a LP count of three (2 mirrors plus original).
Volume Group Descriptor Area(VGDA)	Information about all the LVs and PVs within a VG. The first 64K of a PV is reserved for this area - defined in <sys/bootrecord.h>. The VGDA consists of BOOTRECORD: - first 512 bytes. Allows the Read Only System (ROS) to boot system BAD BLK DIRECTORY - found in <sys/bddir.h> LVM RECORD - found in <lvmrec.h>
Volume Group Status Area(VGSA)	Information about which PPs that are stale and which PVs are missing within a VG. The LVM and SCSI driver reserves somewhere between 7-10% of the available disk space for LVM maps, etc.
Physical Volume ID (PVID)	The PVID is an amalgamation of the machine’s serial number (from the systems EPROMs) and the date that the PVID is being generated. This combination insures theextremely low chance of two disks being created with the same PVID. Finally, when a system is booted, the disk configurator goes and looks at the PVID sitting on each disk platter and then compares that to an entry in ODM. If the entry is found, then the disk is given the hdiskX name that is associated with the ODM entry for the PVID.
Quorum	Quorum is a sort of “sanity” check that LVM uses to resolve possible data confliction and prevent data corruption. Quorum is a method by which 51% or more quorum votes must be available to a volume group before LVM actions can continue. Quorum is issued to a disk in a volume group according to how the disk was created within the volume group. When a volume group consists of one disk, there are two VGDA’s on that disk. Thus, this single disk volume group has a quorum vote of 2. When another disk is added to the volume group with an “extendvg”, then this new disk gets one VGDA, but the original, first disk still retains the two VGDA’s. When the volume group has been extended to three disks, the third disk gets the spare VGDA sitting on the first disk and then each disk has a quorum vote of 1. Every disk after the third disk is automatically given one VGDA, and thus one vote.
Volume Group ID (VGID)	Just as the PVID is a soft serial number for a disk, the VGID is the soft serial number for the volume group. It is this serial number, not the volume group’s ascii name, which all low level LVM commands reference. Additionally, it is the basis for the LVIDs created on that VGID.
Logical Volume Control Block (LVCB)	The logical volume control block (lvcb) consists of the first 512 bytes of a logical volume. This area holds important information such as the creation date of the logical volume, information about mirrored copies, and possible mount points in a journaled filesystem.
Logical Volume ID (LVID)	The LVID is the soft serial number used to represent the logical volume to the LVM libraries and low level commands. The LVID is created from the VGID of the volume group, a decimal point, and a number which represents the order which the logical volume was created on the volume group.

Now for the cheet sheet

Directory and Files
Directories and Files
Tools
diagnostic	diag - used to hot swap the disk cfgmgr - used mak sure the new disk is seen # to add new disk from the scsi0 controller cfgmgr -l scsi0
Create/Remove hard disk	cfgmgr -l scsi0 mkdev -c disk -l <pv> rmdev -dl <pv>
Physical Volumes
display	lspv lspv <pv> (detailed) lspv -l <pv> (list logical volumes) lspv -p <pv> (physical partition usage)
PVID	chdev -l <pv> -a pv=yes chdev -l <pv> -a pv=clear Note: PVID's are automatically added when the disk is placed into a vg
adding	chdev -l <pv> -a pv=yes (new) chpv -v a <pv> (adds back the removed disk)
removing	chpv -v r <pv>
change physical attributes	chpv -a y <pv> (changes allocatable state to YES) chpv -a n <pv> (changes allocatable state to NO)
moving	migratepv <old pv> <new pv>
Volume Groups
display	lsvg lsvg <vg> (detailed) lsvg -l <vg> (list all logical volumes in goup) lsvg -p <vg> (list all physical volumes in group) lsvg -o (lists all varied on) lsvg -M <vg> (lists assicated disks and state) ## Details volume group info for the hard disk lqueryvg -Atp <pv> lqueryvg -p <disk> -v (Determine the VG ID# on disk) lqueryvg -p <disk> -L (Show all the LV ID#/names in the VG on disk) lqueryvg -p <disk> -P (Show all the PV ID# that reside in the VG on disk)
varyon	varyonvg <vg> varyonvg -f <vg> (force) varyonvg -s <vg> (maintenance mode can use VG commands but lv 's cannot be opened for i/o access) varyoffvg <vg> Note: the varyon command activiates the volume goup which means it is available for use
ODM related	## Determine if the ODM and VGDA are correct (in sync) getlvodm -u <vg> ## tries to resync VGDA, LV control blocks and ODM synclvodm <vg> ## If the message 0516-366 lsvg: Volume group <vg> is locked is ever seen putlvodm -K `gtlvodm -v <vg>`
creating	mkvg -y <vg> -s <PP size> <pv> mkvg -y datavg -s 4 hdisk1 Note: the PP size will be the size of the physical partition size you want 4MB, 8MB
extending	extendvg <vg> <pv>
reducing	reducevg -d <vg> <pv> ## removes the PVID from the VGDA when a disk has vanished without using the reducevg command reducevg <vg> <PVID>
removing	varyoffvg <vg> exportvg <vg> Note: the export command nukes everything regardingthe volume goup in the ODM and /etc/filesystems
checking	## check to see if underlying disk has grown in size chvg -g <vg> Note: use this command if you are using SAN LUN's that have increased in size
change volume attributes	## auto vary on a volume at system start chvg -a y # Turns on/off quorum checking on a volume group chvg -Q [y|n] <vg>
renaming	varyoffvg <old vg name> lsvg -p <old vg name> (obtain disk names) exportvg <old vg name> import -y <new vg name> <pv> varyonvg <new vg name> mount -a
importing	importvg -y <vg> <pv> importvg <pv> (will use rootvg as default vg)
exporting	varyoffvg <vg> exportvg <vg> Note: if the volume has an active paging space this must be turned off before
Logical Volumes
display	lslv <lv> lslv -l <lv> (list all physical volumes in logical volume) lslv -m <lv> (list ppartition mapping) ## Display lv control block information getlvcb -AT <lv>
creating	mklv <vg> <# of PP's> <pv> mklv -y <lv name> <vg> <# of PP's> <pv> ## Create a mirrored named logical volume mklv -y <lv> -c <copies 2 or 3> <vg> <# of PP's> <pv> ## create a JFSlog logical Volume mklv -y <lv name> -t jfslog <vg> <# of PP's> <pv>
extending	extendlv <lv> <additonal # of PP's> extendlv <lv> <size of volume in B||M|G>
reducing/resizing	see filesystem below
removing	rmlv <lv>
moving	migratepv -l <lv> <old pv> <new pv>
adding a mirror to a non-mirrored volume	mklvcopy -s n <lv> <copies 2 or 3> <pv>
removing a mirror copy from a mirrored volume	rmlvcopy <lv> <copies 1 or 2> rmlvcopy <lv> <copies 1 or 2> <pv> (specified pv) unmirrorvg <vg> <pv>
synchronize logical volume	syncvg -p <pv> syncvg -v <vg> syncvg -l <lv>
mirror any unmirrored volumes	mirrorvg <vg> <pv>
change volume attributes	## Enable the bad-block relocation policy chlv -b [y|n] <lv>
renaming	chlv -n <new lv name> <old lv name>
Miscellaneous	## Initialises an LV for use as an JFSlog logform </dev/lv>
Filesystems
display	lsfs lsfs -q <fs> (detailed) Note: use the '-q' to see if the logical volume size is bigger than the filesystem size
create	## create new filesystem, -A means to mount after restart crfs -v jfs -d <lv> -m <mountpoint> -A yes ## Create logical volume, filesystem, mountpoint, add entry to /etc/filesystems at the specified size crfs -v jfs2 -g <vg> -m <mountpoint> -a size=<size in 512k blocks|M|G) -A yes Note: there are two types of filesystems jfs and jfs2, jfs2 allows you to decrease the filesystem size , you cannot reduce a jfs filesystem
remove	rmfs <fs> Note: if all filesystems have been removed from a logical volume then the logical volume is removed as well.
resize	chfs -a size=<new size> <fs> chfs -a size=1G /var (specific size, can be used to increase and decrease) chfs -a size=+1G /var (increase by 1GB) chfs -a size=-1GB /var (reduce by 1GB) Note: this will automatically increase or decrease the underlying logical volume as well
freeze/unfreeze	chfs -a freeze=<time in seconds> <fs> chfs -a freeze=off <fs>
split mirrored copy	chfs -a splitcopy=<split copy mountpoint>-a copy=2 <fs> chfs -a splitcopy=/backup -a copy=2 /testfs
change	## Change the mountpoint chfs -m <new mountpoint> <fs> ## Do not mount after a restart chfs -A no <fs> ## Mount read-only chfs -p ro <fs>
mount	mount mount [<fs>|<lv>] mount -a mount all
defrag	defragfs -q <fs> (report defrag status) defragfs -r <fs> (runs in report only mode - no action) defragfs <fs> (actually defrags the filesystem)
checking and repairing	fsck [-y|-n] <fs> (check a filesystem) fsck -p <fs> (restores primary superblock from backup copy if corrupt)
Miscellaneous
Complete VG, LV and FS with mirroring example	## Create the volume group mkvg -s 256 -y datavg hdisk2 ## Create the jfs2 log logical volume and initialize it this for the volume group mklv -t jfs2log -y dataloglv datavg 1 logform /dev/dataloglv ## Create the logical volume mklv -t jfs2 -y data01lv datavg 8 ## Create the filesystems that will use the logical volume crfs -v jfs -d data01lv -m /data01 -A yes ## Add an additional hard disk to the volume group extendvg datavg hdisk3 ## Now mirror both the volume group log logical volume and the logical volume mklvcopy dataloglv 2 mklvcopy data01lv 2 ## Make sure everything is sync'ed both the log and the logical volume syncvg -v datavg ## Make sure everything is OK lsvg -l datavg ## a quick way to perform the above in two steps mklv -c 2 -t jfs2 -y data02lv datavg 8 crfs -v jfs -d data02lv -m /data02 -A yes ## mount everything and check mount -a
Replaced failed mirror drive	## break the mirror (two ways to do this) rmlvcopy <lv name> 1 <broken disk> unmirrorvg <lv> <broken pv > ## remove the disk from the vg reducevg <vgname> <broken pv > ## remove the hdisk from ODM rmdev -dl <broken pv> ## physically replace the disk diag -> function select -> task selection -> hot plug task -> scsi and scsi raid hot plug manager -> replace/remove a device attached to an scsi hot swap enclosure device -> select disk and follow instructions ## configure new disk an check the new number (hopefully the same) cfgmgr -v lsdev -Cc <pv> ## add back to volume group extendvg <vg> <pv> ## create mirror (two ways to do this) mklvcopy <lv> 2 <pv> mirrorvg <lv> ## sync mirror syncvg -l <lv> ## If this is the rootvg there are additonal steps to take bosboot -ad /dev/<pv> bootlist -m normal <pv> <pv> bootlist -m normal -o
Accidently remove a mirrored disk or SAN LUN disappeared off the network	## This procedure places back a mirror disk that you have accidently pulled or that a SAN LUN disappeared off the network ## and its states is classed as "missing" ## see that the disk is in a missing state (see PV state column), also see stale volumes lsvg -p <vg> lsvg -M <vg> ## To make the disk active again we use the varyonvg command varyonvg <vg> ## see that the disk is in a active state (see PV state column) lsvg -p <vg> ## Now re-sync the volumes in that volume group syncvg -v <vg> ## Make sure that no volumes are stale lsvg -M <vg> ## Determine if the ODM and VGDA are correct (in sync) getlvodm -u <vg>

Courtesy by http://www.datadisk.co.uk

AIX System Recovery Tips and Techniques

An AIX recovery can be necessary as a result of a number of events: the loss of some system files, an unexplained system crash, a site environmental problem, or simply a request for a system recovery test. Either way, be prepared to hit the ground running and get the recovery done—or be ready to pack your bags and say goodbye.

AIX recovery is a basic skill; there are no excuses for not having it or not being prepared to use it as part of a disaster recovery (DR) plan. AIX system recovery isn’t rocket science, but you need to have your wits about you. This article will help you prepare to perform a recovery quickly and with confidence.

Prepare, Prepare, Prepare

Key requirements for a successful recovery are an up-to-date configuration listing of the target machine, a current system backup, and application backups or re-installation media. Whether you’re dealing with a full or partial restore, or a simulated or real disaster, the processes involved are the same. If you’re prepared with these prerequisites, your recovery will go smoothly; if not, you’re in for a difficult time.

The best way to ensure that you’re prepared is to routinely (at least weekly) create a system-bootable backup of your AIX servers to capture the sort of periodic changes that occur on a regular basis, such as PTFs and minor file changes. Also, track the status of applications and data being backed up daily, because these components are much more volatile than the OS itself. Typically, application backup is the responsibility of an operational team, but as an AIX systems admin, you should be informed that the data is being backed up successfully; after all, the applications do reside on your machine. You should also take a configuration report for each server. At minimum, this should include the output of the following commands:

• lspv
• lsvg -l <vgname>, lsvg -p <vgname>, lsvg <vgname>(for all volume groups)
• lsslot -c slot
• lscfg -vp
• lsdev
• lsattr -El sys0

A script can collect this information for you automatically and archive it off machine by, for example, emailing its output file to you. With the information these commands provide, you’ll be on a good footing to a confident recovery.

Expect the Unexpected

Recovering a system to a new server at a remote site typically involves restoring the OS from a tape or DVD bootable backup. You can perform a boot restore via the network if you’ve taken remote network system saves with netboots (e.g., Storix or NIM), but this process is much slower than restoring from a tape or DVD, and only the largest “hot site” facilities have netboot host capabilities. The rest of us must make do with bare-bones recovery from the trenches.

The restore-from-bootable-media process is straightforward. First, because it’s best to start up without a network attachment, make sure all Ethernet and other network cables (other than storage) are unplugged. Next, insert the bootable media—tape or DVD—into a boot-capable drive and start the system. It’s best if the server you’re restoring to closely matches the specs of the failed server, but some differences can be accommodated. For example, the root volume group (rootvg) disk(s) might not be the same size, but as long as they’re larger, not smaller, the restore will complete. You should be prepared to alter some of the logical volume copies or re-size the logical volumes during the AIX recovery process if your restore product allows.

Confirm Settings in New Environment

Confirm from the networks team or DR manager what IPs you’ll be using for the following:

• Host and gateway IP addresses (IPv4 and IPv6)
• Subnet mask
• DNS servers
• DNS entries (forward and reverse for all addresses owned by the host)
• Firewall, ipfilter, and/or tcpwrapper rules
• Printed copies of all customized directories showing ownership and permission settings
• Mail relay host (if your machine forwards mail)
• xntpd server

You might be on a different LAN or VLAN for the duration of the disaster, so be sure to document the IP environment for the recovery site so that you’re not fighting network issues during recovery operations. And, of course, if your system interacts with other servers or services, ensure that those are accessible from the recovery site.

Review Operational Parameters

Remember that all Ethernet cables should be disconnected at startup. If the machine comes up with the network interface disabled, that’s good; if it comes up enabled, you forgot to take out the Ethernet cables, which can complicate startup troubleshooting. (You don’t want some automated application process kicking off uncontrolled sessions.) When the AIX recovery boot-up completes, it’s time to check all the operational parameters, and then check them again. Review the /etc/inittab file, comment out any non-required services you don’t want started, then refresh the inittab with telinit -q. Check out root’s crontab and review any non-required periodic jobs that might start. Once you’re satisfied that all application processes and undesired mail sending processes are commented out, stop or kill any processes that might have been kicked off before you reviewed /etc/inittab and crontabs. You might want to delete any outbound queued email files held in /var/spool/mqueue because the mail system might try to send those messages, which you might not want until you’re ready for full production operation.

Next, stop and re-start sendmail so you have a clean mail agent running. Review any firewall, ipfilter, and tcp wrapper rules you have; these will undoubtedly have to be amended now that you’re in recovery mode and in a geographically different environment. If your machine’s database applications use raw devices, be sure to check the ownerships of these devices in /dev, because these likely would have been changed on a system restore. Most databases use async I/O; check that your databases are running using pstat -a. If your machine is on AIX 6.1 or later, database processes are started automatically. On AIX 5.3, you’ll probably need to start them up.

On the Network

Bring the machine onto the network by connecting the Ethernet cables (you should have already configured the net interfaces). Verify that you can ping the network gateway (both IPv4 and IPv6 if you use it), your DNS server, and any necessary collaborative servers. Validate that your configured DNS correctly resolves local and global names, and give special attention to reverse name resolution for the IP addresses owned by the AIX system you’re recovering. One of the most common root causes of startup failure is missing DNS entries for the new network environment.

If static routes are required to reach any internal or WAN networks other than through the default gateway, use the netstat -rn command to verify that the routes exist, and add them if needed. Stop and start the sshd service if it’s present (from the console, or you’ll cut off your command-line session). Test a remote connection, such as Telnet or ssh, to ensure you have remote access capabilities. Next, begin the xntpd service to start getting the machine time synced, and verify it with the date command. You should now be able to send a test email to make sure sendmail forwarding works:

1. echo "test mail" | mail david.tansley@btinternet.com

Now you're ready to configure your data volumes.

Bring In the Disks

Internal data volumes won’t typically be saved with the system bootable backup. You must restore them separately, so be sure your DR plan includes the instructions for this step. If you use a Storage Area Network (SAN), the SAN volumes might reside at a remote site. If so, be sure to get iSCSI or FC zoning correct—there’s no time to mess around—then run cfgmgr to bring them in. The same goes for locally attached disks. Be sure to create your disk raid configuration, if required. If you’re only going to be at DR for a few days, you can generally forgo RAID altogether—the complexity isn’t worth the risk of a disk failure during DR operations.

Create the volume groups and file systems based on the configuration reports you captured previously. It might be advantageous to create a script when you're gathering your reports of the host configurations; this lets you automatically create the file systems and saves you a lot of time, as I’ve learned from experience.

Restore the Application Data

As I noted earlier, your application data must be backed up separately from the bootable OS media, and thus must be restored separately. If you’re using a third-party product for your application backups, check that the client is running and talking back to the remote backup server. Next, restore the applications and the data (if you do incremental backups, ensure the operational team has the full list of tapes required). This is typically the operational team’s responsibility, so be sure to hurry them along. When all the data is recovered, review the permissions of the base directories or file systems, then review them again. Once you’re satisfied, prepare to start up the services in a controlled manner, one by one. If you have databases to restore, make sure you have the latest dumps before restoring them. Review the processes running and consult with the applications’ support teams so that there are no issues. If everything looks good, stop all applications.

A reboot with Pause

Now’s the time to test that the machine can reboot. You might be thinking, “Why do this; let’s just get the machine recovered?” Well, if the machine goes down at a working DR site, it doesn’t reflect well on you or your team, so run this test now before you release the machine to the users. There are many factors that could stop an automatic boot, and because your initial boot was closely attended, you might not have encountered or noticed them. Simple things such as an incorrectly seated Ethernet cable or an IP address conflict can cause a reboot to stop and wait for manual intervention, so a trial reboot is essential.

First, clear the errorlog with errclear 0 so that you have a clean error logging sheet. Issue the bosboot and then the reboot commands. You should always issue a bosboot before any reboot or shut down because it’s a good habit to have. If for some reason the boot hangs, count your lucky stars that you discovered the problem now.

A Final Cross-Check, Please

Once the machine comes back up, check that all services are up. Get the support team to connect to the applications. Then relax and wait for the phone calls to come in on some other tinkering that needs to be done. This is inevitable, I’m afraid; however, the bulk of your work is now done.

Courtesy by PowerITpro