Verifying cluster health

The SC//Reliant™ platform uses Sensu, an open-source monitoring event pipeline, to collect, process, and forward monitoring telemetry. We provide Sensu service checks, which can be used to determine cluster health.

A cluster is considered healthy under the following conditions:

• All nodes in the cluster are online and communicating successfully
• The cluster has quorum
• All resources on the nodes are running
• There are no failed actions on the nodes
• Fencing is enabled, and both fence devices are running
• Data volumes are fully replicated and not resynchronizing

There are two ways to establish that. The service checks give you the monitored view, which is what alerts on and reports health over time. The cluster tools on the node give you the live view, which is what you use while you are working on a cluster and need to see the current state rather than the state at the last check interval. Both are described below.

Running is not the same as protected

A cluster can be serving its workload normally while having no capacity to survive a second problem. This state is worth recognizing on its own, because it looks healthy from the outside and it is the state in which starting further work is most likely to cause an outage. A cluster is running but unprotected when any of the following is true.

• A node is in standby. Its resources have moved to the other node, so the workload is fine, but there is nowhere left to fail over to.
• A data volume is resynchronizing. The copy on the catching-up node is incomplete, so it cannot be promoted until the resync finishes.
• Fencing is disabled, or a fence device is stopped. The cluster cannot isolate a misbehaving node, so it cannot safely recover from one.
• A resource has accumulated failures on a node. It may be barred from running there, which removes a failover target without anything appearing to be down.

Do not begin planned work on a cluster in any of these states. Return it to a fully protected state first, or accept that a failure during the window means manual recovery rather than automatic failover. See Performing maintenance on a node.

Checking cluster health on a node

When you are connected to a node, the cluster tools report the current state directly. This is the quickest way to confirm the conditions above, and it does not wait for a check interval.

Connect to the node using SSH. Every command in this section runs on the node rather than on your own computer.

ssh <node_hostname>

For example, ssh sit0001red0. For the shell access prerequisites and more detail on connecting, see Performing maintenance on a node.

Nodes, resources, and failed actions

The following command reports these three together. The -1 flag prints the state once and exits, and -r includes resources that are stopped, which are otherwise hidden.

sudo crm_mon -1 -r

Read the output as follows.

• All nodes online. Both nodes appear on the Online: line. A node that is in standby is listed separately as Node <hostname>: standby rather than under Online:, and a node still evacuating reads standby (with active resources).
• Quorum held. The second line reads partition with quorum. If it reads partition WITHOUT quorum, the node cannot see enough of the cluster to make decisions safely.
• All resources running. No resource in the full list reads Stopped. Bear in mind that both nodes legitimately run resources at all times. Clone sets run an instance per node, the DRBD resource runs as Master on one node and Slave on the other, and each node hosts the fence device belonging to its partner.
• No failed actions. A healthy cluster prints no Failed Actions section at all. If one appears, each entry names the resource, the operation that failed, and the node it failed on.

Quorum

Quorum determines whether a node is allowed to run resources at all. On a two-node cluster it deserves a closer look than the single line in crm_mon, because the arithmetic is unusual.

sudo corosync-quorumtool -s

A healthy two-node cluster reports Quorate: Yes, with Expected votes: 2, Total votes: 2, and Quorum: 1. The Flags line reads 2Node Quorate WaitForAll.

Those values are deliberate rather than a misconfiguration. A quorum of 1 means a single surviving node keeps running the workload instead of stopping, which is the behavior you want when one node fails. WaitForAll means that after a full cluster restart both nodes must be seen once before resources start, which prevents each node from independently deciding it is the survivor. The cluster property no-quorum-policy completes the picture, and on this platform it is set to ignore, meaning the cluster continues to run resources rather than stopping them if quorum is ever lost.

sudo pcs property show

NOTE: rvc cluster info reports No Quorum Policy as Yes or No, which does not convey the configured policy. Read the actual value with sudo pcs property show, where no-quorum-policy reports ignore, stop, freeze, or suicide.

Corosync communicates over a ring, and a fault on the ring can partition the cluster even while both nodes are powered on and reachable by other means.

sudo corosync-cfgtool -s

A healthy ring reports status = ring 0 active with no faults.

Fencing

A cluster that cannot fence cannot safely recover from a node that stops responding, so fencing is part of health rather than a separate concern. Two things have to be true.

rvc cluster info

In the Option column, Stonith must read Enabled. This reports cluster-wide state from either node, along with whether the cluster is in maintenance mode and a per-node table of standby and unclean status. Maintenance Mode reads Yes or No.

Both fence devices must also be running, which rvc cluster info does not show. Each node hosts the device that fences its partner, so a cluster with a node in standby has one device stopped and cannot fence the surviving node.

sudo crm_mon -1 -r

Look for one stonith:external/ipmi device per node, each Started on the opposite node from the one it fences.

Resource failures

A resource that has failed repeatedly on a node can be barred from running there, which quietly removes a failover target while everything still appears to be running.

sudo pcs resource failcount show

A healthy cluster reports No failcounts. The same information appears as a Migration Summary section, empty for each node, in the following.

sudo crm_mon -1 --failcounts

Failure records persist after a resource recovers, so a cluster can keep reporting failed actions long after the underlying problem is fixed. Once you have investigated and resolved the cause, clear the history so the cluster reflects current reality. Omit the resource ID to clear every resource.

sudo pcs resource cleanup <resource_id>

IMPORTANT: Clear failure records only after you understand why the failure happened. The records are the evidence, and clearing them to make a dashboard look healthy discards the only history of an intermittent fault.

Replication

cat /proc/drbd

This reports replication state, which the cluster tools do not show in detail. A healthy volume reads cs:Connected with ds:UpToDate/UpToDate and oos:0. Run it on both nodes, because the output describes replication from the perspective of the node you run it on.

A volume that reads cs:SyncSource or cs:SyncTarget, with oos above zero, is resynchronizing. The cluster is serving normally but has no complete second copy until it finishes. For help reading either state, see DRBD replication status reference and Data volumes and DRBD.

Service checks for cluster health

Five service checks report on the conditions described above. On a healthy cluster all five return OK.

For what each check does in detail, the exact text it returns in every state, its interval, and the thresholds that trigger a warning or a critical, see Monitoring check reference. That page is the authoritative source for check behavior. This page covers how to read the checks when your goal is to establish whether a cluster is healthy.

How service checks behave during maintenance

Planned work sets several checks to a failing state, and this is expected rather than a fault. Placing a node in standby stops its cluster-managed resources, including its DRBD instance, so both cluster_maintenance and drbd report CRITICAL for the length of the window. Checks can also report UNKNOWN briefly while a node is converging, for example during a package update that restarts a service the check depends on. Each returns to OK on the first interval after the node is back in service.

Because alerts can be routed to downstream ticketing systems, carry out cluster maintenance inside a planned change window so the alerts are expected. See Performing maintenance on a node.

Retrieving healthcheck details

To retrieve the details of each Sensu check, use the SC//Reliant API. You specify the hostname of the node, and you can optionally filter by check name and by status.

The examples below use the Ruby helper script, which runs on your own computer rather than on a node in the cluster. You need API access on your Platform Manager account before any of them work. For the script itself, see API examples using Ruby, and for the access and authentication requirements, see API access requirements.

The Ruby script is one of several ways to call the API. If you would rather work in PowerShell, or make calls directly from a browser without any local tooling, see API examples using PowerShell and API examples using Swagger. The request, the parameters, and the response are the same whichever you choose.

In the examples, replace <customer_code> with your customer code, which is the prefix your node hostnames share. For a node named sit0001red0, the customer code is sit. Replace the hostname with the node you want details for.

The following example retrieves the status of the cluster_maintenance, corosync_resources, corosync_node, and drbd service checks.

ruby api.rb <customer_code> get api/nodes/sit0001red0/checks\?filter_by_name=cluster_maintenance,corosync_resources,corosync_node,drbd | jq '.checks[] | {name: .name, output: .output}'

The jq filter reduces the response to the check name and its output text. Omit it to see the full response, which returns the following for each check.

The status field and the filter_by_status parameter both use the following codes.

In the following example, you are querying all failing service checks for the sit0001red0 node, which is every status other than OK.

ruby api.rb <customer_code> get api/nodes/sit0001red0/checks\?filter_by_status=1,2,3

The history array is worth reading when you are investigating something that has already recovered, because it shows when a check began failing and when it returned to OK without you having to consult a separate monitoring system.

For the full request syntax, all supported parameters, and the response schema, see Retrieve checks for a single node. To query several nodes in one request, for example every node in a node group, see Retrieve checks for multiple nodes.

Related documentation

• Performing maintenance on a node: standby, fencing, and the verification steps that follow maintenance.
• DRBD replication status reference: reading replication state in detail.
• Pacemaker and Corosync (high availability): the cluster software, quorum, STONITH, and the failover process.
• Data volumes and DRBD: how replication works between the two nodes.
• Operating the cluster: maintenance mode, standby, and how the two controls interact.
• Monitoring check reference: every service check available on the platform, not only the cluster ones.
• Monitoring your environment: how Sensu collects events and forwards them to downstream destinations.
• Retrieve checks for a single node: the API endpoint used on this page.
• API access requirements: prerequisites for calling the API.