How to cleanup AIX EMC ODM definitions

From powerlink.emc.com:

1. Before making any changes, collect host logs to document the current configuration. At a minimum, save the following: inq, lsdev -Cc disk, lsdev -Cc adapter, lspv, and lsvg
2. Shutdown the application(s), unmount the file system(s), and varyoff all volume groups except for rootvg. Do not export the volume groups.

   # varyoffvg <vg_name>

   Check with lsvg -o (confirm that only rootvg is varied on)
   If no PowerPath, skip all steps with power names.
3. For CLARiiON configuration, if Navisphere Agent is running, stop it:

   # /etc/rc.agent stop

4. Remove paths from Powerpath configuration:

   # powermt remove hba=all

5. Delete all hdiskpower devices:

   # lsdev -Cc disk -Fname | grep power | xargs -n1 rmdev -dl

6. Remove the PowerPath driver instance:

   # rmdev -dl powerpath0

7. Delete all hdisk devices:
   For Symmetrix devices, use this command:

   # lsdev -CtSYMM* -Fname | xargs -n1 rmdev -dl

   For CLARiiON devices, use this command:

   # lsdev -CtCLAR* -Fname | xargs -n1 rmdev -dl

8. Confirm with lsdev -Cc disk that there are no EMC hdisks or hdiskpowers.
9. Remove all Fiber driver instances:

   # rmdev -Rdl fscsiX

   (X being driver instance number, i.e. 0,1,2, etc.)
10. Verify through lsdev -Cc driver that there are no more fiber driver instances (fscsi).
11. Change the adapter instances in Defined state

    # rmdev -l fcsX

    (X being adapter instance number, i.e. 0,1,2, etc.)
12. Create the hdisk entries for all EMC devices:

    # emc_cfgmgr

    or

    # cfgmgr -vl fcsx

    (x being each adapter instance which was rebuilt). Skip this part if no PowerPath.
13. Configure all EMC devices into PowerPath:

    # powermt config

14. Check the system to see if it now displays correctly:

    # powermt display
    # powermt display dev=all
    # lsdev -Cc disk
    # /etc/rc.agent start

Netapp Snapmirror Setup Guide

Snapmirror is an licensed utility in Netapp to do data transfer across filers. Snapmirror works at Volume level or Qtree level. Snapmirror is mainly used for disaster recovery and replication.

Snapmirrror needs a source and destination filer. (When source and destination are the same filer, the snapmirror happens on local filer itself.  This is when you have to replicate volumes inside a filer. If you need DR capabilities of a volume inside a filer, you have to try syncmirror ).

Synchronous SnapMirror is a SnapMirror feature in which the data on one system is replicated on another system at, or near, the same time it is written to the first system. Synchronous SnapMirror synchronously replicates data between single or clustered storage systems situated at remote sites using either an IP or a Fibre Channel connection. Before Data ONTAP saves data to disk, it collects written data in NVRAM. Then, at a point in time called a consistency point, it sends the data to disk.

When the Synchronous SnapMirror feature is enabled, the source system forwards data to the destination system as it is written in NVRAM. Then, at the consistency point, the source system sends its data to disk and tells the destination system to also send its data to disk.

This guides you quickly through the Snapmirror setup and commands.

1) Enable Snapmirror on source and destination filer

source-filer> options snapmirror.enable
snapmirror.enable            on
source-filer>
source-filer> options snapmirror.access
snapmirror.access            legacy
source-filer>

2) Snapmirror Access
Make sure destination filer has snapmirror access to the source filer. The snapmirror filer's name or IP address should be in /etc/snapmirror.allow. Use wrfile to add entries to /etc/snapmirror.allow.

source-filer> rdfile /etc/snapmirror.allow
destination-filer
destination-filer2
source-filer>

3) Initializing a Snapmirror relation

Volume snapmirror : Create a destination volume on destination netapp filer, of same size as source volume or greater size. For volume snapmirror, the destination volume should be in restricted mode. For example, let us consider we are snapmirroring a 100G volume - we create the destination volume and make it restricted.

destination-filer> vol create demo_destination aggr01 100G
destination-filer> vol restrict demo_destination

 

 

Volume SnapMirror creates a Snapshot copy before performing the initial transfer. This copy is referred to as the baseline Snapshot copy. After performing an initial transfer of all data in the volume, VSM (Volume SnapMirror) sends to the destination only the blocks that have changed since the last successful replication. When SnapMirror performs an update transfer, it creates another new Snapshot copy and compares the changed blocks. These changed blocks are sent as part of the update transfer.

Snapmirror is always destination filer driven. So the snapmirror initialize has to be done on destination filer. The below command starts the baseline transfer.

destination-filer> snapmirror initialize -S source-filer:demo_source destination-filer:demo_destination
Transfer started.
Monitor progress with 'snapmirror status' or the snapmirror log.
destination-filer>

Qtree Snapmirror : For qtree snapmirror, you should not create the destination qtree. The snapmirror command automatically creates the destination qtree. So just volume creation of required size is good enough.

 

Qtree SnapMirror determines changed data by first looking through the inode file for inodes that have changed and changed inodes of the interesting qtree for changed data blocks. The SnapMirror software then transfers only the new or changed data blocks from this Snapshot copy that is associated with the designated qtree. On the destination volume, a new Snapshot copy is then created that contains a complete point-in-time copy of the entire destination volume, but that is associated specifically with the particular qtree that has been replicated.

destination-filer> snapmirror initialize -S source-filer:/vol/demo1/qtree destination-filer:/vol/demo1/qtree
Transfer started.
Monitor progress with 'snapmirror status' or the snapmirror log.
4) Monitoring the status : Snapmirror data transfer status can be monitored either from source or destination filer. Use "snapmirror status" to check the status.

destination-filer> snapmirror status
Snapmirror is on.
Source                          Destination                          State          Lag Status
source-filer:demo_source        destination-filer:demo_destination   Uninitialized  -   Transferring (1690 MB done)
source-filer:/vol/demo1/qtree   destination-filer:/vol/demo1/qtree   Uninitialized  -   Transferring (32 MB done)
destination-filer>

5) Snapmirror schedule : This is the schedule used by the destination filer for updating the mirror. It informs the SnapMirror scheduler when transfers will be initiated. The schedule field can either contain the word sync to specify synchronous mirroring or a cron-style specification of when to update the mirror. The cronstyle schedule contains four space-separated fields.

If you want to sync the data on a scheduled frequency, you can set that in destination filer's /etc/snapmirror.conf . The time settings are similar to Unix cron. You can set a synchronous snapmirror schedule in /etc/snapmirror.conf by adding “sync” instead of the cron style frequency.

destination-filer> rdfile /etc/snapmirror.conf
source-filer:demo_source        destination-filer:demo_destination - 0 * * *  # This syncs every hour
source-filer:/vol/demo1/qtree   destination-filer:/vol/demo1/qtree - 0 21 * * # This syncs every 9:00 pm
destination-filer>

6) Other Snapmirror commands

• To break snapmirror relation - do snapmirror quiesce and snapmirror break.
• To update snapmirror data  - do snapmirror update
• To resync a broken relation - do snapmirror resync.
• To abort a relation - do snapmirror abort

Snapmirror do provide multipath support. More than one physical path between a source and a destination system might be desired for a mirror relationship. Multipath support allows SnapMirror traffic to be load balanced between these paths and provides for failover in the event of a network outage.

To read how to tune the performance & speed of the netapp snapmirror or snapvault replication transfers and adjust the transfer bandwidth , go to Tuning Snapmirror & Snapvault replication data transfer speed

SAN Migration via LVM: Don't Forget Raw Logical Volumes

Need to migrate several AIX logical partitions to a new SAN storage subsystem. When we looked at our options, we originally planned to migrate the root volume group (rootvg) using a mksysb backup, which would let us restore onto the new SAN and test the reboot without impacting the original configuration. The data volume groups were to be migrated using the AIX Logical Volume Manager (LVM). But our plan ran into a hitch when we realized the source system had some raw logical volumes in rootvg.

Well, what difference would that make? 

If you have raw logical volumes (LVs) in rootvg, the contents of those LVs won't get backed up by the mksysb command. The reason for this is that the mksysb file-system image is in backup-file format. In other words mksysb only captures files that are present in a mounted file system. 

So we were left with a challenge. How could we migrate the data in the raw LVs to the new SAN? 

We thought of using the alt_disk_copy command to clone rootvg to a disk on the new SAN, but then we would have faced the same difficulty as with the mksysb command. The alt_disk_copy command backs up only mounted file systems -- not raw logical volumes. Another option was to create a separate application-level backup of the raw LVs. Unfortunately, that would involve some extra backups and possibly more extensive downtime. 

Mirror, Mirror! 

In the end we found a simple solution that didn't involve rebuilding or restoring the raw logical volumes. We created a mirror of the rootvg using the mirrorvg command. This approach guaranteed we'd have a mirror copy of rootvg, including all the raw logical volumes. The volume group on the new SAN would be fully synchronised with the copy on the original SAN. Mirrorvg gave us a truly up-to-date mirror of the rootvg. Once the mirror was completed, we'd change the bootlist to point to the physical volume on the new SAN, and then reboot. 

The migration went smoothly, and after the mirrorvg, we rebuilt the boot image on the new SAN disk using the bosboot command. 

We then rebooted and were able to mirror the rest of the data volume groups to the new SAN. Finally, we removed the copies from the original SAN disk, starting with rootvg (via the unmirrorvg command). We had to remove the original rootvg disk from the bootlist, and finally detach the original SAN from the AIX VM. 

Playing in the SAN
There were some pros and cons with using a mirror instead of a mksysb backup or disk clone. With the mksysb restore or alt_disk_copy, you can verify ahead of time that the OS will boot neatly on the new SAN. You can also play a little with the new SAN, checking performance and getting famlliar with procedures for assigning new disk. 

The risk is that you would lose any rootvg changes between the time of creating the backup/clone and the time you cut over to the new SAN. 

The mirrorvg isn't a perfect solution either. For starters, you don't get a chance to verify the VM boots from the new SAN until you have shut down your system on the original SAN. Another limitation is that you can only mirror to the new SAN if your configuration allows you to connect to the old SAN and the new SAN at the same time. That may not be an option for your environment. 

Quick and Clean
In my client's environment, the mirror of the volume group worked cleanly and quickly. Most importantly, it captured all of the data that was in the raw logical volumes and mirrored them all to the new SAN subsystem. 

In preparing our migration, we were able to draw on a number of excellent resources. One of these was an article by my colleague, Chris Gibson, on Using the AIX Logical Volume Manager to Perform SAN Storage Migrations. That article includes many practical examples, as well as the commands you need to run to make your SAN migration as smooth as possible. 

Resources 

AIX Installation and Migrationhttp://bit.ly/OmKmLH 

Clone a running system to an alternate disk using alt_disk_copyhttp://bit.ly/QudX0S 

Create a backup of the root volume group using mksysb command 
http://bit.ly/OUxngf 

Mirror all the logical volumes in a volume group with mirrorvghttp://bit.ly/MSWR1H 

Give rootvg the space it needs: why your OS needs some room to breathehttp://ibm.co/Oe745W 


Courtesy POWER IT Pro.

Helpful SAN LUN Disk AIX Commands

When adding LUN’s to AIX, the below commands can be helpful to add SAN disks.  These commands will help add the disks and also assist in identifying an issue as to why the server may not see the newly create SAN disks.

Get WWN, Model, Part Number, etc. From HBA

lscfg -vp -l fcs0

Rescan all AIX devices after LUN Assignment
cfgmgr

Identify Fiber Channel Adapters

lsdev -Cc adapter | grep fcs

Get Fiber Card Status

fcstat fcs0

Check if HBA port is up or down

fcstat -D fcs0 | grep Attention

See All Local and SAN Disks on System

lsdev -Cc disk