Operating the cluster

The SC//Reliant™ platform creates high-availability clusters using Pacemaker and Corosync. This document walks you through common cluster operations for administrators.

Most routine failures in a cluster are handled autonomously, without operator action. A workload that fails on one node migrates to the surviving node automatically, governed by the Pacemaker and Corosync behavior described on this page. For the wider set of conditions the platform remediates on its own, see Monitoring your environment. The operations on this page are the ones you perform deliberately, when you need that automation to stand aside.

The following are some clustering concepts to understand prior to performing cluster operations.

Understanding maintenance mode and standby

Maintenance mode and standby are two separate controls that both change how Pacemaker manages resources, but they solve different problems. The distinction comes down to scope and intent:

Maintenance mode applies to the whole cluster and suspends resource management. Pacemaker stops starting, stopping, moving, and monitoring resources anywhere in the cluster. Workloads keep running exactly where they are, and nothing fails over.
Standby applies to a single node and evacuates it. The node becomes ineligible to run resources, so Pacemaker gracefully stops the workloads on that node and starts them on the other node in the cluster.

In short, maintenance mode freezes the cluster in place, while standby moves work off one node. Maintenance mode is what you use when Pacemaker's automation would get in the way of what you are doing. Standby is what you use when you need one node emptied so you can work on it.

Comparing maintenance mode and standby

	Maintenance mode	Standby
Scope	The entire cluster	A single node
Effect on resource management	Suspended cluster-wide. Pacemaker does not start, stop, or move resources.	Remains active. Pacemaker continues to manage resources normally on the rest of the cluster.
Effect on running workloads	Workloads keep running where they are and are not moved.	Workloads on the node are gracefully stopped and restarted on the other node.
Automatic failover	Disabled	Still enabled
Resource monitoring	Suspended	Still active
Node eligibility for new resources	Unchanged	The node is no longer eligible to run resources
Use it when	Pacemaker's automatic reaction to a change would interfere with the work you are doing, such as system maintenance or troubleshooting that would otherwise trigger a failover.	You need to take one node out of service and want its workloads running elsewhere first, such as operating system updates or hardware repair.
Set with	The SC//Reliant API or the RVC CLI	`pcs` on the node

Not every task needs either control. Updating container image tags to deploy a new version of a container does not require maintenance mode, and a Puppet run to deploy general configuration changes usually does not either. For advice on whether a specific action requires maintenance mode, reach out to SC//Reliant Support or your account manager.

How maintenance mode and standby interact

Because standby depends on Pacemaker to move resources and maintenance mode suspends exactly that behavior, the two controls conflict when combined. What happens when you place a node on standby depends on whether the cluster is in maintenance mode.

When the cluster is not in maintenance mode (normal operation):

Pacemaker sees that you want to take the node out of service.
It immediately consults its policy engine and starts the process of gracefully stopping all virtual machines and containers on that node. If the graceful shutdown of a virtual machine is unsuccessful, there is a grace period of 30 seconds before a virtual machine is forcefully shut off.
It then starts those resources on another available node in the cluster.
Result: The node is marked standby and its resources are actively migrated elsewhere. This is the standard procedure for performing maintenance on a single node. It also effectively marks the node the resources are being migrated to as the new primary node.

When the cluster is in maintenance mode:

The command succeeds and the node's status is officially marked as standby.
However, because the entire cluster is in maintenance mode, Pacemaker does NOT move the resources.
The resources that were running on the node continue to run there, even though the node is marked as standby. Pacemaker's policy engine is effectively suspended and does not act on the state change.
Result: The node is flagged as standby, but the resources are stranded on it until maintenance mode is turned off. Once you turn maintenance mode off, Pacemaker wakes up, sees that the node is in standby, and begins moving the resources off.

IMPORTANT: If your goal is to evacuate a node, confirm the cluster is not in maintenance mode before placing the node on standby. Otherwise the node reports standby while still running its workloads, which can be mistaken for a completed evacuation.

Using maintenance mode

The following describes how to toggle maintenance mode for a cluster. To check whether a cluster is currently in maintenance mode, use rvc cluster maintenance-mode? or read the maintenance field in the output of rvc cluster status.

Turning on maintenance mode

Using the SC//Reliant API

The following syntax can be used to turn on maintenance mode for a node using the API. For more details on the syntax, see Turn on maintenance mode for a target.

ruby api.rb rte POST api/v2/jobs/rvc '{"method":"cluster.enable_maintenance_mode","hostnames":["sit0001red0"]}'

Using the SC//Reliant Virtual Controller 3.x CLI

The following syntax can be used to turn on maintenance mode for a node using the RVC CLI. For more details on the syntax, see cluster.enable_maintenance_mode.

rvc cluster enable-maintenance-mode

Turning off maintenance mode

Using the SC//Reliant API

The following syntax can be used to turn off maintenance mode for a node using the API. For more details on the syntax, see Turn off maintenance mode for a target.

ruby api.rb rte POST api/v2/jobs/rvc '{"method":"cluster.disable_maintenance_mode","hostnames":["sit0017red0"]}'

Using the SC//Reliant Virtual Controller 3.x CLI

The following syntax can be used to turn off maintenance mode for a node using the RVC CLI. For more details on the syntax, see cluster.disable_maintenance_mode.

rvc cluster disable-maintenance-mode

Using standby

Standby is set on the node itself with the pcs utility. To check whether a node is currently on standby, read the standby field in the output of rvc cluster status.

Placing a node on standby

Confirm the cluster is not in maintenance mode, so that Pacemaker is able to migrate the resources off the node.

rvc cluster maintenance-mode?

Place the node on standby.

sudo pcs node standby <node_hostname>

Monitor the cluster as the resources migrate to the other node.

sudo crm_mon

Resources outside the cluster are not managed by Pacemaker and are therefore unaffected by standby. You need to shut those down gracefully yourself.

Removing a node from standby

When you are ready to return the node to service, remove it from standby. The node becomes eligible to run resources again.

sudo pcs node unstandby <node_hostname>

For the full procedure covering primary and secondary nodes, including the DRBD checks to run first, see Performing maintenance on a node.

Additional cluster operations

The following provide additional instruction regarding cluster operations.

For procedures that take a node or the cluster out of service, see Performing maintenance and updates.