Pacemaker and Corosync (high availability)

Pacemaker and Corosync (high availability)

Pacemaker and Corosync are two open-source software components that work together to create the high-availability (HA) clusters used by the SC//Reliant™ platform. They ensure that critical services and applications remain operational even if hardware or software failures occur on a node, and they coordinate the STONITH fencing mechanism that protects data integrity during failover.

Role of each component

  • Corosync, the cluster communication layer. Corosync provides the foundational communication and membership services for the cluster: managing cluster membership, ensuring reliable message delivery between nodes, and maintaining quorum to prevent split-brain scenarios.
  • Pacemaker, the cluster resource manager (CRM). Pacemaker sits on top of Corosync and is responsible for managing the cluster's resources (applications, services, IP addresses, filesystems, and so on) to keep them available. Its responsibilities include resource management, failover orchestration, policy enforcement, and fencing (STONITH) to prevent data corruption.

Together, Pacemaker and Corosync form a tightly coupled duo for achieving high availability. Corosync handles low-level cluster communication and membership, and Pacemaker manages resources and orchestrates failover.

Operational resilience

Resilience in the SC//Reliant platform is designed to hold up under stress. When the Wide Area Network (WAN) is down, when the management plane is unreachable, or when disaster recovery (DR) fires, controls continue to operate at the same standard as steady state. Cluster-level high availability is one part of that posture. The platform's fail-secure offline mode, DR security parity, and immutable security logging are the others.

Fail-secure offline mode

The security controls applied to the SC//Reliant platform Edge Device are platform-wide and remain in effect regardless of management-plane connectivity, per Scale Computing's documented Encryption and Cryptography Policy, Network Security Policy, Endpoint Hardening Policy, and Media Handling Policy.

The Edge Device is designed for continuous local operation. When connectivity to the cloud-hosted management plane is interrupted, the device continues to enforce its locally cached Puppet-managed configuration, retains its OpenVPN and Puppet X.509 certificate-based mutual authentication state, and continues to apply local authorization controls (RBAC, default-deny iptables, hardened CIS/NIST/SANS-aligned Debian baseline).

Disaster recovery and business continuity

Scale Computing maintains formal, documented disaster recovery procedures reviewed at least annually, owned by the Chief Technology Officer (CTO) function and validated against the Business Continuity Plan. DR and failover environments are deployed from controlled, version-managed configuration that mirrors production. They inherit the same access controls, MFA requirements, encryption standards, logging, and network segmentation as production, and are validated as part of the annual Business Continuity and Disaster Recovery (BCDR) exercise. No control downgrades occur during recovery.

Recovery Time Objective (RTO) and Recovery Point Objective (RPO) targets are defined in the Business Continuity Plan, and HA cluster patterns at the edge (described in the STONITH sections below) support local workload continuity independent of management-plane availability.

Immutable security logging

Security logs are protected through architectural separation and platform-level integrity controls. Edge-node and management-plane events are forwarded in real time to the customer's chosen Security Information and Event Management (SIEM) platform, hosted in an environment separate from the source systems, so an actor on a source system cannot retroactively alter logs that have already been centralized. The platform is SIEM-agnostic: any syslog-capable SIEM the customer already operates can be the destination. Immutability, indexing, and tamper-evidence behavior at the destination depend on the SIEM platform the customer selects.

Access to the SIEM is governed by RBAC with separation of duties between operational administrators and auditors, and is reviewed quarterly per PCI DSS. Management-plane API activity is captured by the native audit-log capability of the environment where each customer's management plane is hosted, for example, AWS CloudTrail when hosted on AWS, Azure Activity Log on Azure, Google Cloud Audit Logs on Google Cloud, or the equivalent audit facility of an on-premises or private-cloud deployment. The platform is cloud-agnostic and does not require any specific provider's audit tooling. A loss-of-logging alert fires if the central log host indexes no events for a 30-minute window. Audit logs are retained for at least 365 days, with file integrity monitoring on log servers and policy-level prohibition on modification or deletion documented in the Security Monitoring and Detection Policy.

A deeper dive into STONITH

STONITH (Shoot The Other node In The Head) is a crucial component of high-availability clusters, designed to prevent data corruption and ensure system integrity in the event of a node failure or communication breakdown.

In a high-availability cluster, multiple nodes work together to provide continuous service. If one node becomes unresponsive, the cluster needs to determine if that node is truly down or if it is merely experiencing network isolation (a split-brain scenario). In a split-brain situation, both nodes might incorrectly assume they are the primary and attempt to access and modify shared resources (such as shared storage) simultaneously. This can lead to severe data corruption.

How STONITH works

STONITH acts as a fencing mechanism, isolating a misbehaving or unresponsive node from shared resources or completely powering it off. The goal is to ensure that only one healthy node has access to the critical shared data at any given time.

The STONITH method used on SC//Reliant platform edge devices is power fencing, the most commonly used method. A STONITH device (typically an intelligent power strip, an IPMI interface, or a management interface on a hypervisor) is used to physically cut power to the unresponsive node, forcing it into a known offline state.

Why STONITH is necessary

  • Data integrity. STONITH absolutely prevents split-brain scenarios and the resulting data corruption that would occur if multiple nodes tried to write to the same shared data simultaneously.
  • Resource management. If a service or resource fails to stop cleanly on a node, STONITH ensures the entire node is taken offline so the resource can be safely started on a healthy node.
  • Guaranteed isolation. Because a node being unresponsive to the cluster software does not necessarily mean it has stopped running processes or accessing data, STONITH provides a definitive way to ensure the node is completely isolated.

STONITH is the last-resort safety net in an HA cluster, ensuring that even in the most problematic failure scenarios, data integrity is preserved by decisively isolating or eliminating the problematic node.

Cluster components

  • Nodes. The two physical or virtual servers in the cluster.
  • Operating system. The base OS running on each node (Debian Linux on the SC//Reliant platform).
  • Cluster software (Pacemaker / Corosync). Corosync provides the underlying communication and membership layer. Pacemaker is the cluster resource manager.
  • Heartbeat / cluster communication. A dedicated network link (often redundant) used by Corosync for nodes to send "I'm alive" messages and coordinate cluster operations.
  • DRBD resource. DRBD replicates a block device from the active (primary) node to the passive (secondary) node in real time and presents a virtual block device (for example, /dev/drbdX) to the operating system.
  • DRBD replication link. A dedicated, high-speed network connection between the two nodes specifically for DRBD traffic. This is important for performance and to avoid interference with other network traffic.
  • STONITH. A critical fencing mechanism. If a node becomes unresponsive, STONITH ensures it is truly powered down or isolated before the other node takes over its resources. It is typically implemented using out-of-band management interfaces such as IPMI on servers or smart Power Distribution Units (PDUs). The active node can instruct these devices to power off the faulty node.
  • Out-of-band management. The hardware or network path used for STONITH. This path must be independent of the regular data and cluster communication networks.
  • VIP (Virtual IP Address). A floating IP address that clients use to access the application service. Pacemaker manages this IP address and assigns it to the currently active node. If the active node fails, Pacemaker moves the VIP (and the application service) to the surviving node.

High-level failover process

  1. Failure detection. The cluster software (Corosync/Pacemaker) on the surviving node detects that the partner node is unresponsive (for example, missed heartbeats).
  2. STONITH action. Pacemaker on the surviving node initiates a STONITH action against the unresponsive node. This typically involves sending a command via IPMI to power off the unresponsive node, ensuring it is definitely down and will not cause data corruption.
  3. Resource takeover. Once the unresponsive node is confirmed to be fenced (powered down or isolated):
    • Pacemaker promotes the local DRBD resource to primary.
    • It mounts the filesystem on /dev/drbdX.
    • It starts the application services.
    • It assigns the Virtual IP to the surviving node's network interface.
  4. Service resumption. Clients can now connect to the VIP, which directs them to the surviving node, and the application service continues running with its data intact.

This setup provides high availability by ensuring that if one node fails, the other can quickly and safely take over its responsibilities with minimal downtime and no data loss (assuming synchronous DRBD replication).

© 2026 Scale Computing, Inc. All rights reserved.

Scale Computing, SC//AcuVigil, SC//Connect, and SC//Reliant are trademarks of Scale Computing, Inc.