HACMP

To check Disk Heartbeats
lssrc -ls topsvcs

To check cspoc log
cat /tmp/cspoc.log |egrep -p "FAIL|ERROR" |more

To check clverify log grep
"Check:" /var/hacmp/clverify/clverify.log |sort |uniq

To check clutil log for last 2 days
tail -n 2 /var/hacmp/log/clutils.log

To check the status of the cluster
lssrc -ls clstrmgrES

To check the cluster related services at one shot
lssrc -a|egrep "svcs|ES"

To identify the RG movement activities
tail -f /tmp/hacmp.out

To identify the RG location
clRGinfo

To monitor the status of HACMP in a terminal
clstat -a

SNMP based tool to show cluster state
cldump

similar to cldump, perl script to show cluster state.
cldisp

list the local view of the cluster topology.
cltopinfo

list the local view of the cluster subsystems.
clshowsrv -a

locate the resource groups and display status.
clfindres (-s)

rotate some of the log files.
clcycle

a cluster ping program with more arguments.
cl_ping

cluster rsh program that take cluster node names as argument.
clrsh

which nodes are active?
clgetactivenodes

what is the name of the local node?
get_local_nodename

check the HACMP ODM.
clconfig

online/offline or move resource groups.
clRGmove

sync/fix the cluster.
cldare

Sync the cluster topology, resources and run the verification
cldare -rtV normal

list the resource groups.
cllsgrp

create a large snapshot of the hacmp configuration.
clsnapshotinfo

list the network configuration of an hacmp cluster.
cllscf

show the resource group configuration.
clshowres

show network interface information.
cllsif

show short resource group information.
cllsres

ls clstrmgrES list the cluster manager state.
lssrc

ls topsvcs show heartbeat information.
lssrc

list a node centric overview of the hacmp configuration.
cllsnode

When using qha you need to understand the cluster manager internal states. Here’s a description of the possible cluster states:
ST_INIT: cluster configured and down
ST_JOINING: node joining the cluster
ST_VOTING: Inter-node decision state for an event
ST_RP_RUNNING: cluster running recovery program
ST_BARRIER: clstrmgr waiting at the barrier statement
ST_CBARRIER: clstrmgr is exiting recovery program
ST_UNSTABLE: cluster unstable
NOT_CONFIGURED: HA installed but not configured
RP_FAILED: event script failed
ST_STABLE: cluster services are running with managed resources (stable cluster) or cluster services have been "forced" down with resource groups potentially in the UNMANAGED state (HACMP 5.4 only)

clcomdES
(Secure remote command execution) "Cluster communication enhanced security/usr/sbin/cluster/etc/rhosts --- to accept incoming communication from clcomdES (cluster communucation enahanced security) /usr/es/sbin/cluster/etc/rhosts
The clcomdES is used for cluster configuration operations such as cluster synchronisation
cluster management (C-SPoC) * Dynamic re-configuration DARE configuration. (DARE ) operation"

cllockd
For concurrent access

HACMP log files
/tmp/hacmp.out:
It records the output generated by the event scripts as they execute. When checking the /tmp/hacmp.out file, search for EVENT FAILED messages. These messages indicate that a failure has occurred. Then, starting from the failure message, read back through the log file to determine exactly what went wrong.

The /tmp/hacmp.out file is a standard text file. The system creates a new hacmp.out log file every day and retains the last seven copies. Each copy is identified by a number appended to the file name. The most recent log file is named /tmp/hacmp.out; the oldest version of the file is named /tmp/hacmp.out.7

/usr/es/adm/cluster.log: It is the main HACMP log file. HACMP error messages and messages about HACMP-related events are appended to this log with the time and date at which they occurred

/usr/es/sbin/cluster/history/cluster.mmddyyyy: It contains time-stamped, formatted messages generated by HACMP scripts. The system creates a cluster history file whenever cluster events occur, identifying each file by the file name extension mmddyyyy, where mm indicates the month, dd indicates the day, and yyyy indicates the year.

/tmp/cspoc.log: It contains time-stamped, formatted messages generated by HACMP C-SPOC commands. The /tmp/cspoc.log file resides on the node that invokes the C-SPOC command.