How to Add a New Resource Group to an Active Cluster
Contents
1. Introduction
This article describes how to add a new resource group »RES_GRP_03« to an HACMP cluster. Primary node for the new resource group shall be »queen«. We find the following situation:
queen# clRGinfo
-----------------------------------------------------------------------------
Group Name Group State Node
-----------------------------------------------------------------------------
RES_GRP_01 ONLINE king
OFFLINE queen
RES_GRP_02 ONLINE king
OFFLINE queen
Conclusion: There are two resource groups - both online on king.
2. Create new Volume Groups
First we need to scan for new disks:
queen# cfgmgr
and set a PVID to all new disks:
queen# chdev -l hdiskA1 -a pv=yes
queen# chdev -l hdiskA2 -a pv=yes
queen# chdev -l hdiskB1 -a pv=yes
:
:
and change the reservation policy for use with HACMP:
queen# chdev -l hdiskA1 -a reserve_policy=no_reserve
queen# chdev -l hdiskA2 -a reserve_policy=no_reserve
queen# chdev -l hdiskB1 -a reserve_policy=no_reserve
:
:
Now we are ready to run the configuration manager on the other node and set the reservation policy of the disks:
king# cfgmgr
king# chdev -l hdiskA1 -a reserve_policy=no_reserve
king# chdev -l hdiskA2 -a reserve_policy=no_reserve
king# chdev -l hdiskB1 -a reserve_policy=no_reserve
:
:
Since the new resource group will be primarily online on queen, we create the volume group and the filesystems on queen. Anyway, there is no technical reason for doing this on the primary node. We could do this on the other node as well. The volume groups will be created as follows:
queen# mkvg -C -V 130 -S -s 32 -y RG03data hdiskA1 hdiskA2 hdiskB1 hdiskB2
queen# varyonvg RG03data
queen# chvg -Qn RG03data
Please note: The volume groups we create are enhanced-concurrent-capable. By default these volume groups have the following properties: Quorum is on, auto-varyon is off. Further note the volume group's major number we set with the »-V« option. Get free major numbers on both nodes with the »lvlstmajor« command.
With the volume groups created and vary'd on we can go on creating logical volumes. Usually we would use logical volumes mirrored over two datacenters. So we ensure that all first LV copies (PV1 in »lslv -m« output) reside in Datacenter A and all second LV copies (PV2 in »lslv -m« output) reside in Datacenter B. We use superstrictness and the most narrow upper bound possible to reduce the risk of a mirroring mess, e.g.
queen# mklv -t jfs2 -e x -a e -u 2 -s s -c 2 -y lvrg03fs01 RG03data 256 hdiskA1 hdiskB1 hdiskA2 hdiskB2
where hdiskA1 and hdiskA2 reside in datacenter A and hdiskB1 and hdiskB2 reside in datacenter B.
Filesystems on top of our logical volumes should be created with the mount = false option, e.g.
queen# crfs -v jfs2 -m /filesystem/rg3/1 -d /dev/lvrg03fs01 -A no -p rw -a agblksize=512 -a logname=INLINE
queen# mount /filesystem/rg3/1
queen# crfs -v jfs2 -m /filesystem/rg3/2 -d /dev/lvrg03fs02 -A no -p rw -a agblksize=512 -a logname=INLINE
queen# mount /filesystem/rg3/1
Once all filesystems are created we unmount everything we just created and close the volume group:
queen# umount /filesystem/rg3/1
queen# umount /filesystem/rg3/2
queen# varyoffvg RG03data
Now we are ready to import all Volume Groups we created into the other node.
king# importvg -n -V -y
Get information about major number, Volume Group, and PVID from the other node.
3. Add Resources: New Application Start/Stop Scripts
For the new application server we first need application start and stop scripts. We assume that we've got these scripts and copied them to /etc/hacmp on queen. Then we can add a new application server:
queen# smitty hacmp
-> Extended Configuration
-> Extended Resource Configuration
-> HACMP Extended Resources Configuration
-> Configure HACMP Applications
-> Configure HACMP Application Servers
-> Add an Application Server
Add Application Server
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Server Name [app_srv03]
* Start Script [/etc/hacmp/start_srv03]
* Stop Script [/etc/hacmp/stop_srv03]
Application Monitor Name(s) +
Since the scripts are local we have to copy them over to king:
queen# scp -p /etc/hacmp/st*_srv03 king:/etc/hacmp/
4. Add Resources: New Service IP Adresses
For the new Resource Group one or more service ip addresses are needed. First we have to add the new service ip addresses to /etc/hosts on both nodes:
# vi /etc/hosts
111.111.111.130 haservice3 haservice3.domain.com
111.111.111.131 haservice31 haservice31.domain.com
Then we let HACMP discover information:
queen# smitty hacmp
-> Extended Configuration
-> Discover HACMP-related Information from Configured Nodes
Then we have to define the new IP labels as a HACMP resource.
queen# smitty hacmp
-> Extended Configuration
-> Extended Resource Configuration
-> HACMP Extended Resources Configuration
-> Configure HACMP Service IP Labels/Addresses
-> Add a Service IP Label/Address
+--------------------------------------------------------------------------+
| Select a Service IP Label/Address type |
| |
| Move cursor to desired item and press Enter. |
| |
| Configurable on Multiple Nodes |
| Bound to a Single Node |
| |
| F1=Help F2=Refresh F3=Cancel |
| F8=Image F10=Exit Enter=Do |
| /=Find n=Find Next |
+--------------------------------------------------------------------------+
The service address needs to move with the application - so we select "Configurable on Multiple Nodes" here.
Add a Service IP Label/Address configurable on Multiple Nodes (extended)
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* IP Label/Address haservice3 +
Netmask(IPv4)/Prefix Length(IPv6) []
* Network Name net_ether_01
Alternate HW Address to accompany IP Label/Address []
5. Create a New Resource Group
queen# smitty hacmp
-> Extended Configuration
-> Extended Resource Configuration
-> HACMP Extended Resource Group Configuration
-> Add a Resource Group
Add a Resource Group (extended)
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Resource Group Name [RES_GRP_03]
* Participating Nodes (Default Node Priority) [queen king] +
Startup Policy Online On Home Node Only +
Fallover Policy Fallover To Next Priority Node In The List +
Fallback Policy Never Fallback +
If you pick the partitipating nodes with from the list, you might end up with th wrong order. The home node must be the first in the line!
Don't be confused by the clRGinfo output:
queen# clRGinfo
-----------------------------------------------------------------------------
Group Name Group State Node
-----------------------------------------------------------------------------
RES_GRP_01 ONLINE king
OFFLINE queen
RES_GRP_02 ONLINE king
OFFLINE queen
clRGinfo: Operation failed, error = 0.
clRGinfo will show the new resource group once we synchronized the cluster.
6. Add the Resources to the New Resource Group
Before we start to add resources to our new resource group it might be a good idea to synchronize the cluster first.
The last step is to add resources to our resource groups. If you didn't run the HACMP discovering tool yet, you have to run it now. Otherwise HACMP can't find our newly created volume groups. If you exactly did the steps in the order of this article then the discovering tool has already be run.
This said, we add the resources:
queen# smitty hacmp
-> Extended Configuration
-> Extended Resource Configuration
-> HACMP Extended Resource Group Configuration
-> Change/Show Resources and Attributes for a Resource Group
+--------------------------------------------------------------------------+
| Change/Show Resources and Attributes for a Resource Group |
| |
| Move cursor to desired item and press Enter. |
| |
| RES_GRP_01 |
| RES_GRP_02 |
| RES_GRP_03 |
| |
| F1=Help F2=Refresh F3=Cancel |
| F8=Image F10=Exit Enter=Do |
| /=Find n=Find Next |
+--------------------------------------------------------------------------+
Change/Show All Resources and Attributes for a Custom Resource Group
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[TOP] [Entry Fields]
Resource Group Name [RES_GRP_03]
Participating Nodes (Default Node Priority) [queen king] +
Startup Policy Online On Home Node Only +
Fallover Policy Fallover To Next Priority Node In The List +
Fallback Policy Never Fallback +
Service IP Labels/Addresses [haservice3 hasevice31] +
Application Servers [app_srv03] +
Volume Groups [RG03data] +
Use forced varyon of volume groups, if necessary true +
Automatically Import Volume Groups false +
Filesystems (empty is ALL for VGs specified) [] +
Filesystems Consistency Check logredo +
Filesystems Recovery Method sequential +
Filesystems mounted before IP configured false +
Filesystems/Directories to Export [] +
+
Filesystems/Directories to NFS Mount []
Network For NFS Mount [] +
[MORE...10]
7. Final Synchronization of the Cluster
Now synchronize the cluster...
queen# smitty hacmp
-> Extended Configuration
-> Extended Verification and Synchronization
and clRGinfo shows three resource groups:
queen# clRGinfo
-----------------------------------------------------------------------------
Group Name Group State Node
-----------------------------------------------------------------------------
RES_GRP_01 ONLINE king
OFFLINE queen
RES_GRP_02 ONLINE king
OFFLINE queen
RES_GRP_03 ONLINE queen
OFFLINE king
You will also notice that the service IP addresses have been acquired and that all our filesystems have been mounted on the primary node.
Contents
- Introduction
- Create new Volume Groups
- Add Resources: New Service IP Adresses
- Add Resources: New Application Start/Stop Scripts
- Create a New Resource Group
- Add the Resources to the New Resource Group
- Final Synchronization of the Cluster
- Related Information
1. Introduction
This article describes how to add a new resource group »RES_GRP_03« to an HACMP cluster. Primary node for the new resource group shall be »queen«. We find the following situation:
queen# clRGinfo
-----------------------------------------------------------------------------
Group Name Group State Node
-----------------------------------------------------------------------------
RES_GRP_01 ONLINE king
OFFLINE queen
RES_GRP_02 ONLINE king
OFFLINE queen
Conclusion: There are two resource groups - both online on king.
2. Create new Volume Groups
First we need to scan for new disks:
queen# cfgmgr
and set a PVID to all new disks:
queen# chdev -l hdiskA1 -a pv=yes
queen# chdev -l hdiskA2 -a pv=yes
queen# chdev -l hdiskB1 -a pv=yes
:
:
and change the reservation policy for use with HACMP:
queen# chdev -l hdiskA1 -a reserve_policy=no_reserve
queen# chdev -l hdiskA2 -a reserve_policy=no_reserve
queen# chdev -l hdiskB1 -a reserve_policy=no_reserve
:
:
Now we are ready to run the configuration manager on the other node and set the reservation policy of the disks:
king# cfgmgr
king# chdev -l hdiskA1 -a reserve_policy=no_reserve
king# chdev -l hdiskA2 -a reserve_policy=no_reserve
king# chdev -l hdiskB1 -a reserve_policy=no_reserve
:
:
Since the new resource group will be primarily online on queen, we create the volume group and the filesystems on queen. Anyway, there is no technical reason for doing this on the primary node. We could do this on the other node as well. The volume groups will be created as follows:
queen# mkvg -C -V 130 -S -s 32 -y RG03data hdiskA1 hdiskA2 hdiskB1 hdiskB2
queen# varyonvg RG03data
queen# chvg -Qn RG03data
Please note: The volume groups we create are enhanced-concurrent-capable. By default these volume groups have the following properties: Quorum is on, auto-varyon is off. Further note the volume group's major number we set with the »-V« option. Get free major numbers on both nodes with the »lvlstmajor« command.
With the volume groups created and vary'd on we can go on creating logical volumes. Usually we would use logical volumes mirrored over two datacenters. So we ensure that all first LV copies (PV1 in »lslv -m« output) reside in Datacenter A and all second LV copies (PV2 in »lslv -m« output) reside in Datacenter B. We use superstrictness and the most narrow upper bound possible to reduce the risk of a mirroring mess, e.g.
queen# mklv -t jfs2 -e x -a e -u 2 -s s -c 2 -y lvrg03fs01 RG03data 256 hdiskA1 hdiskB1 hdiskA2 hdiskB2
where hdiskA1 and hdiskA2 reside in datacenter A and hdiskB1 and hdiskB2 reside in datacenter B.
Filesystems on top of our logical volumes should be created with the mount = false option, e.g.
queen# crfs -v jfs2 -m /filesystem/rg3/1 -d /dev/lvrg03fs01 -A no -p rw -a agblksize=512 -a logname=INLINE
queen# mount /filesystem/rg3/1
queen# crfs -v jfs2 -m /filesystem/rg3/2 -d /dev/lvrg03fs02 -A no -p rw -a agblksize=512 -a logname=INLINE
queen# mount /filesystem/rg3/1
Once all filesystems are created we unmount everything we just created and close the volume group:
queen# umount /filesystem/rg3/1
queen# umount /filesystem/rg3/2
queen# varyoffvg RG03data
Now we are ready to import all Volume Groups we created into the other node.
king# importvg -n -V
Get information about major number, Volume Group, and PVID from the other node.
3. Add Resources: New Application Start/Stop Scripts
For the new application server we first need application start and stop scripts. We assume that we've got these scripts and copied them to /etc/hacmp on queen. Then we can add a new application server:
queen# smitty hacmp
-> Extended Configuration
-> Extended Resource Configuration
-> HACMP Extended Resources Configuration
-> Configure HACMP Applications
-> Configure HACMP Application Servers
-> Add an Application Server
Add Application Server
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Server Name [app_srv03]
* Start Script [/etc/hacmp/start_srv03]
* Stop Script [/etc/hacmp/stop_srv03]
Application Monitor Name(s) +
Since the scripts are local we have to copy them over to king:
queen# scp -p /etc/hacmp/st*_srv03 king:/etc/hacmp/
4. Add Resources: New Service IP Adresses
For the new Resource Group one or more service ip addresses are needed. First we have to add the new service ip addresses to /etc/hosts on both nodes:
# vi /etc/hosts
111.111.111.130 haservice3 haservice3.domain.com
111.111.111.131 haservice31 haservice31.domain.com
Then we let HACMP discover information:
queen# smitty hacmp
-> Extended Configuration
-> Discover HACMP-related Information from Configured Nodes
Then we have to define the new IP labels as a HACMP resource.
queen# smitty hacmp
-> Extended Configuration
-> Extended Resource Configuration
-> HACMP Extended Resources Configuration
-> Configure HACMP Service IP Labels/Addresses
-> Add a Service IP Label/Address
+--------------------------------------------------------------------------+
| Select a Service IP Label/Address type |
| |
| Move cursor to desired item and press Enter. |
| |
| Configurable on Multiple Nodes |
| Bound to a Single Node |
| |
| F1=Help F2=Refresh F3=Cancel |
| F8=Image F10=Exit Enter=Do |
| /=Find n=Find Next |
+--------------------------------------------------------------------------+
The service address needs to move with the application - so we select "Configurable on Multiple Nodes" here.
Add a Service IP Label/Address configurable on Multiple Nodes (extended)
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* IP Label/Address haservice3 +
Netmask(IPv4)/Prefix Length(IPv6) []
* Network Name net_ether_01
Alternate HW Address to accompany IP Label/Address []
5. Create a New Resource Group
queen# smitty hacmp
-> Extended Configuration
-> Extended Resource Configuration
-> HACMP Extended Resource Group Configuration
-> Add a Resource Group
Add a Resource Group (extended)
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Resource Group Name [RES_GRP_03]
* Participating Nodes (Default Node Priority) [queen king] +
Startup Policy Online On Home Node Only +
Fallover Policy Fallover To Next Priority Node In The List +
Fallback Policy Never Fallback +
If you pick the partitipating nodes with
Don't be confused by the clRGinfo output:
queen# clRGinfo
-----------------------------------------------------------------------------
Group Name Group State Node
-----------------------------------------------------------------------------
RES_GRP_01 ONLINE king
OFFLINE queen
RES_GRP_02 ONLINE king
OFFLINE queen
clRGinfo: Operation failed, error = 0.
clRGinfo will show the new resource group once we synchronized the cluster.
6. Add the Resources to the New Resource Group
Before we start to add resources to our new resource group it might be a good idea to synchronize the cluster first.
The last step is to add resources to our resource groups. If you didn't run the HACMP discovering tool yet, you have to run it now. Otherwise HACMP can't find our newly created volume groups. If you exactly did the steps in the order of this article then the discovering tool has already be run.
This said, we add the resources:
queen# smitty hacmp
-> Extended Configuration
-> Extended Resource Configuration
-> HACMP Extended Resource Group Configuration
-> Change/Show Resources and Attributes for a Resource Group
+--------------------------------------------------------------------------+
| Change/Show Resources and Attributes for a Resource Group |
| |
| Move cursor to desired item and press Enter. |
| |
| RES_GRP_01 |
| RES_GRP_02 |
| RES_GRP_03 |
| |
| F1=Help F2=Refresh F3=Cancel |
| F8=Image F10=Exit Enter=Do |
| /=Find n=Find Next |
+--------------------------------------------------------------------------+
Change/Show All Resources and Attributes for a Custom Resource Group
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[TOP] [Entry Fields]
Resource Group Name [RES_GRP_03]
Participating Nodes (Default Node Priority) [queen king] +
Startup Policy Online On Home Node Only +
Fallover Policy Fallover To Next Priority Node In The List +
Fallback Policy Never Fallback +
Service IP Labels/Addresses [haservice3 hasevice31] +
Application Servers [app_srv03] +
Volume Groups [RG03data] +
Use forced varyon of volume groups, if necessary true +
Automatically Import Volume Groups false +
Filesystems (empty is ALL for VGs specified) [] +
Filesystems Consistency Check logredo +
Filesystems Recovery Method sequential +
Filesystems mounted before IP configured false +
Filesystems/Directories to Export [] +
+
Filesystems/Directories to NFS Mount []
Network For NFS Mount [] +
[MORE...10]
7. Final Synchronization of the Cluster
Now synchronize the cluster...
queen# smitty hacmp
-> Extended Configuration
-> Extended Verification and Synchronization
and clRGinfo shows three resource groups:
queen# clRGinfo
-----------------------------------------------------------------------------
Group Name Group State Node
-----------------------------------------------------------------------------
RES_GRP_01 ONLINE king
OFFLINE queen
RES_GRP_02 ONLINE king
OFFLINE queen
RES_GRP_03 ONLINE queen
OFFLINE king
You will also notice that the service IP addresses have been acquired and that all our filesystems have been mounted on the primary node.
