HACMP - How to Add a New Resource Group to an Active Cluster

How to Add a New Resource Group to an Active Cluster


Contents


  1. Introduction
  2. Create new Volume Groups
  3. Add Resources: New Service IP Adresses
  4. Add Resources: New Application Start/Stop Scripts
  5. Create a New Resource Group
  6. Add the Resources to the New Resource Group
  7. Final Synchronization of the Cluster
  8. Related Information



1. Introduction

This article describes how to add a new resource group »RES_GRP_03« to an HACMP cluster. Primary node for the new resource group shall be »queen«. We find the following situation:

queen# clRGinfo
-----------------------------------------------------------------------------
Group Name     Group State                  Node        
-----------------------------------------------------------------------------
RES_GRP_01     ONLINE                       king      
                           OFFLINE                      queen      

RES_GRP_02     ONLINE                       king      
                           OFFLINE                      queen      
Conclusion: There are two resource groups - both online on king.



2. Create new Volume Groups

First we need to scan for new disks:

queen# cfgmgr
and set a PVID to all new disks:

queen# chdev -l hdiskA1 -a pv=yes
queen# chdev -l hdiskA2 -a pv=yes
queen# chdev -l hdiskB1 -a pv=yes
                      :
                      :
and change the reservation policy for use with HACMP:

queen# chdev -l hdiskA1 -a reserve_policy=no_reserve
queen# chdev -l hdiskA2 -a reserve_policy=no_reserve
queen# chdev -l hdiskB1 -a reserve_policy=no_reserve
                      :
                      :
Now we are ready to run the configuration manager on the other node and set the reservation policy of the disks:

king# cfgmgr
king# chdev -l hdiskA1 -a reserve_policy=no_reserve
king# chdev -l hdiskA2 -a reserve_policy=no_reserve
king# chdev -l hdiskB1 -a reserve_policy=no_reserve
                      :
                      :
Since the new resource group will be primarily online on queen, we create the volume group and the filesystems on queen. Anyway, there is no technical reason for doing this on the primary node. We could do this on the other node as well. The volume groups will be created as follows:

queen# mkvg -C -V 130 -S -s 32 -y RG03data hdiskA1 hdiskA2 hdiskB1 hdiskB2
queen# varyonvg RG03data
queen# chvg -Qn RG03data
Please note: The volume groups we create are enhanced-concurrent-capable. By default these volume groups have the following properties: Quorum is on, auto-varyon is off. Further note the volume group's major number we set with the »-V« option. Get free major numbers on both nodes with the »lvlstmajor« command.

With the volume groups created and vary'd on we can go on creating logical volumes. Usually we would use logical volumes mirrored over two datacenters. So we ensure that all first LV copies (PV1 in »lslv -m« output) reside in Datacenter A and all second LV copies (PV2 in »lslv -m« output) reside in Datacenter B. We use superstrictness and the most narrow upper bound possible to reduce the risk of a mirroring mess, e.g.

queen# mklv -t jfs2 -e x -a e -u 2 -s s -c 2 -y lvrg03fs01 RG03data 256 hdiskA1 hdiskB1 hdiskA2 hdiskB2
where hdiskA1 and hdiskA2 reside in datacenter A and hdiskB1 and hdiskB2 reside in datacenter B.

Filesystems on top of our logical volumes should be created with the mount = false option, e.g.

queen# crfs -v jfs2 -m /filesystem/rg3/1 -d /dev/lvrg03fs01 -A no -p rw -a agblksize=512 -a logname=INLINE
queen# mount /filesystem/rg3/1
queen# crfs -v jfs2 -m /filesystem/rg3/2 -d /dev/lvrg03fs02 -A no -p rw -a agblksize=512 -a logname=INLINE
queen# mount /filesystem/rg3/1
Once all filesystems are created we unmount everything we just created and close the volume group:

queen# umount /filesystem/rg3/1
queen# umount /filesystem/rg3/2
queen# varyoffvg RG03data
Now we are ready to import all Volume Groups we created into the other node.

king# importvg -n -V -y
Get information about major number, Volume Group, and PVID from the other node.



3. Add Resources: New Application Start/Stop Scripts

For the new application server we first need application start and stop scripts. We assume that we've got these scripts and copied them to /etc/hacmp on queen. Then we can add a new application server:

queen# smitty hacmp
-> Extended Configuration
   -> Extended Resource Configuration
      -> HACMP Extended Resources Configuration
         -> Configure HACMP Applications
            -> Configure HACMP Application Servers
               -> Add an Application Server

                             Add Application Server

Type or select values in entry fields.
Press Enter AFTER making all desired changes.
                                                        [Entry Fields]
* Server Name                                        [app_srv03]
* Start Script                                       [/etc/hacmp/start_srv03]
* Stop Script                                        [/etc/hacmp/stop_srv03]
  Application Monitor Name(s)                                                         +
Since the scripts are local we have to copy them over to king:

queen# scp -p /etc/hacmp/st*_srv03 king:/etc/hacmp/


4. Add Resources: New Service IP Adresses

For the new Resource Group one or more service ip addresses are needed. First we have to add the new service ip addresses to /etc/hosts on both nodes:

# vi /etc/hosts
111.111.111.130 haservice3 haservice3.domain.com
111.111.111.131 haservice31 haservice31.domain.com
Then we let HACMP discover information:

queen# smitty hacmp
-> Extended Configuration
   -> Discover HACMP-related Information from Configured Nodes
Then we have to define the new IP labels as a HACMP resource.

queen# smitty hacmp
-> Extended Configuration
   -> Extended Resource Configuration
      -> HACMP Extended Resources Configuration
         -> Configure HACMP Service IP Labels/Addresses
            -> Add a Service IP Label/Address

  +--------------------------------------------------------------------------+
  |                  Select a Service IP Label/Address type                                   |
  |                                                                                                                 |
  | Move cursor to desired item and press Enter.                                            |
  |                                                                                                                 |
  |   Configurable on Multiple Nodes                                                             |
  |   Bound to a Single Node                                                                         |
  |                                                                                                                |
  | F1=Help                 F2=Refresh              F3=Cancel                               |
  | F8=Image                F10=Exit                Enter=Do                                  |
  | /=Find                  n=Find Next                                                                |
  +--------------------------------------------------------------------------+
The service address needs to move with the application - so we select "Configurable on Multiple Nodes" here.

    Add a Service IP Label/Address configurable on Multiple Nodes (extended)

Type or select values in entry fields.
Press Enter AFTER making all desired changes.

                                                        [Entry Fields]
* IP Label/Address                                    haservice3               +
  Netmask(IPv4)/Prefix Length(IPv6)                  []
* Network Name                                        net_ether_01
  Alternate HW Address to accompany IP Label/Address []



5. Create a New Resource Group

queen# smitty hacmp
-> Extended Configuration
   -> Extended Resource Configuration
      -> HACMP Extended Resource Group Configuration
         -> Add a Resource Group

                        Add a Resource Group (extended)

Type or select values in entry fields.
Press Enter AFTER making all desired changes.

                                                        [Entry Fields]
* Resource Group Name                                [RES_GRP_03]
* Participating Nodes (Default Node Priority)        [queen king]                              +

  Startup Policy                                      Online On Home Node Only                   +
  Fallover Policy                                     Fallover To Next Priority Node In The List +
  Fallback Policy                                     Never Fallback                             +
If you pick the partitipating nodes with from the list, you might end up with th wrong order. The home node must be the first in the line!

Don't be confused by the clRGinfo output:

queen# clRGinfo
-----------------------------------------------------------------------------
Group Name     Group State                  Node        
-----------------------------------------------------------------------------
RES_GRP_01     ONLINE                       king      
                            OFFLINE                      queen      


RES_GRP_02     ONLINE                       king      
                           OFFLINE                      queen      
clRGinfo: Operation failed, error = 0.
clRGinfo will show the new resource group once we synchronized the cluster.



6. Add the Resources to the New Resource Group

Before we start to add resources to our new resource group it might be a good idea to synchronize the cluster first.

The last step is to add resources to our resource groups. If you didn't run the HACMP discovering tool yet, you have to run it now. Otherwise HACMP can't find our newly created volume groups. If you exactly did the steps in the order of this article then the discovering tool has already be run.

This said, we add the resources:

queen# smitty hacmp
-> Extended Configuration
   -> Extended Resource Configuration
      -> HACMP Extended Resource Group Configuration
         -> Change/Show Resources and Attributes for a Resource Group

  +--------------------------------------------------------------------------+
  |        Change/Show Resources and Attributes for a Resource Group         |
  |                                                                          |
  | Move cursor to desired item and press Enter.                             |
  |                                                                          |
  |   RES_GRP_01                                                             |
  |   RES_GRP_02                                                             |
  |   RES_GRP_03                                                             |
  |                                                                          |
  | F1=Help                 F2=Refresh              F3=Cancel                |
  | F8=Image                F10=Exit                Enter=Do                 |
  | /=Find                  n=Find Next                                      |
  +--------------------------------------------------------------------------+

      Change/Show All Resources and Attributes for a Custom Resource Group

Type or select values in entry fields.
Press Enter AFTER making all desired changes.

[TOP]                                                   [Entry Fields]
  Resource Group Name                                [RES_GRP_03]

  Participating Nodes (Default Node Priority)        [queen king]                              +

  Startup Policy                                      Online On Home Node Only                   +
  Fallover Policy                                     Fallover To Next Priority Node In The List +
  Fallback Policy                                     Never Fallback                                         +

  Service IP Labels/Addresses                        [haservice3 hasevice31]                     +
  Application Servers                                [app_srv03]                                             +

  Volume Groups                                      [RG03data]                                              +
  Use forced varyon of volume groups, if necessary    true                                         +
  Automatically Import Volume Groups                  false                                           +

  Filesystems (empty is ALL for VGs specified)       []                                              +
  Filesystems Consistency Check                       logredo                                           +
  Filesystems Recovery Method                         sequential                                      +
  Filesystems mounted before IP configured            false                                         +
  Filesystems/Directories to Export                  []                                                    +
                                                                                                                            +
  Filesystems/Directories to NFS Mount               []
  Network For NFS Mount                              []                                                    +
[MORE...10]


7. Final Synchronization of the Cluster

Now synchronize the cluster...

queen# smitty hacmp
   -> Extended Configuration
     -> Extended Verification and Synchronization
and clRGinfo shows three resource groups:

queen# clRGinfo
-----------------------------------------------------------------------------
Group Name     Group State                  Node        
-----------------------------------------------------------------------------
RES_GRP_01     ONLINE                       king      
                           OFFLINE                      queen      

RES_GRP_02     ONLINE                       king      
                           OFFLINE                      queen      

RES_GRP_03     ONLINE                       queen    
                           OFFLINE                      king    
You will also notice that the service IP addresses have been acquired and that all our filesystems have been mounted on the primary node.