Monitoring check reference

Monitoring check reference

This page lists the service checks that the SC//Reliantâ„¢ platform runs on Linux and Windows hosts, and describes what each one reports. For an overview of how the platform uses Sensu, see Sensu (monitoring).

The following describes each of the available service checks. They can be turned on and off, as well as configured, in SC//Reliant Platform Manager.

Service check reference (Linux hosts)

Service Check Name Description Status Examples Default Interval (seconds)

Free Disk Space

disk_free

Monitors the free disk space on essential mount points including /, /tmp, /home, /var, /var/log, and mounted DRBD devices (from profiles 8.3.8 or newer).

It generates an alert if the free disk space falls below the configured threshold. The default value for a WARNING status is 20%. The default value for CRITICAL status is 10%.

DISK OK - free space:

[OK ] 48% /

[ OK ] 99% /tmp

[ OK ] 48% /home

[ OK ] 48% /var

[ OK ] 66% /var/log

[ OK ] 48% /var/lib/docker

| /=48%;20;10;0;100 /tmp=99%;20;10;0;100 /home=48%;20;10;0;100 /var=48%;20;10;0;100 /var/log=66%;20;10;0;100 /var/lib/docker=48%;20;10;0;100

300

DISK WARN - free space:

[WARN] 15% /

[ OK ] 99% /tmp

[WARN] 18% /home

[WARN] 12% /var

[CRIT] 0% /var/log

| /=15%;20;10;0;100 /tmp=99%;20;10;0;100 /home=18%;20;10;0;100 /var=12%;20;10;0;100 /var/log=0%;20;10;0;100

DISK CRIT - free space:

[CRIT] 0% /

[ OK ] 99% /tmp

[CRIT] 0% /home

[CRIT] 0% /var

[CRIT] 0% /var/log

| /=0%;20;10;0;100 /tmp=99%;20;10;0;100 /home=0%;20;10;0;100 /var=0%;20;10;0;100 /var/log=0%;20;10;0;100

Puppet Client

puppet_client

Monitors the status of a Puppet client and triggers an alert if it has been disabled for longer than the configured threshold.

OK

7200

WARN $reason

CRITICAL reason not specified

CRITICAL $reason

Puppet Errors

puppet_errors

Monitors the last Puppet run. A critical alert is triggered if any failures are detected.

OK: Puppet is still running

OK: Puppet last run had $failcount failures

3600

CRITICAL: Puppet did not start

CRITICAL: Puppet last run had $failcount failures

SSH Server

ssh_server

Monitors the SSH server to verify it is running.

SSH OK - $ssh_server_version (protocol $ssh_protocol) | time=$elapsed_time;;;0.0;$socket_timeout

300

SSH CRITICAL - $ssh_server_version (protocol $ssh_protocol) version mismatch, expected $remote_version

CPU Load Utilization

cpu_utilization

Monitors the CPU load average, a metric that measures the number of processes actively using or waiting for CPU time. A value of 1 indicates that, on average, the CPU is fully utilized. For a system with multiple cores, this value is per-core, so a load of 4 on a quad-core system means each core is fully utilized.

A WARNING is triggered at a per-core load average of 2, and a CRITICAL alert is triggered at 3 or higher. While occasional spikes in load are a normal part of system operation, a consistently high load over time is a strong indicator of a performance bottleneck or a runaway process that requires immediate attention.

This check's load average is calculated over 1, 5, and 15-minute intervals to provide a comprehensive view of the system's performance trend over time.

OK - load average: $load1_average, $load5_average, $load15_average|load1=$average;$warning;$critical;0, load5=$average;$warning;$critical;0, load15=$average;$warning;$critical;0,

300

WARNING - load average: $load1_average, $load5_average, $load15_average|load1=$average;$warning;$critical;0, load5=$average;$warning;$critical;0, load15=$average;$warning;$critical;0,

CRITICAL - load average: $load1_average, $load5_average, $load15_average|load1=$average;$warning;$critical;0, load5=$average;$warning;$critical;0, load15=$average;$warning;$critical;0,

Memory Utilization

memory_available

Monitors the percentage of system memory currently in use.

The default value for a WARNING is 90% utilization, and for a CRITICAL alert is 95% utilization.

CheckMem OK - value = $memory_used | system_memory_used=$memory_used

60

CheckMem WARNING - value = $memory_used | system_memory_used=$memory_used

CheckMem CRITICAL - value = $memory_used | system_memory_used=$memory_used

Corosync Resources

corosync_resources

This check monitors the health of clustered resources managed by Pacemaker and Corosync. This check verifies the status of each configured resource, such as virtual IP addresses, file systems, and application services. It will alert if a resource is offline, failed, or not running on its designated node, which indicates a cluster failure or a resource that has failed to migrate correctly. This check is crucial for ensuring high availability of all critical clustered services.

OK: All resources are up

120

CRITICAL: The resource(s) $resource_list down

Corosync Node

corosync_node

This check monitors the health and connectivity of the nodes within the Corosync cluster quorum. This check ensures that all nodes are communicating and are active members of the cluster. A failure indicates that a node has lost its connection, a critical state that can lead to split-brain scenarios and a loss of high availability for services managed by Pacemaker.

OK: All nodes are up

120

WARNING - The node(s) $node down

DRBD Volumes

drbd

This check monitors the health and replication status of DRBD volumes. This check verifies that the block devices are in a connected, healthy state and are correctly mirroring data between the primary and secondary nodes. A failure indicates a replication issue, such as a disconnected state, which means the volume is no longer highly available and could result in data loss during a failover.

DRBD OK: $device_list_with_status

DRBD OK: Device 50 Connected UpToDate

120

CRITICAL: $device_list_with_status

Host Ports

port_check

This check verifies that a specific network port on the host is open and listening for connections. This check is crucial for ensuring that a service (such as a web server, database, or a custom application) is running and accessible on its designated port. A failed check could mean the service is down, misconfigured, or blocked by a firewall.

This check can be configured to specify the host(s) and port(s) to verify. There is a Strict mode that, when turned on, returns a CRITICAL alert if any port in the check is unresponsive.

OK: All the checked ports on the host are open

300

WARNING: Some of the checked ports on the host are open

CRITICAL: All the checked ports on the host are closed.

When Strict mode is turned on, the check returns critical if any port in the check is closed.

Ajbprocess

ajbprocess

This check monitors the FIPay service and its associated processes to ensure the core payment application is running. FIPay is the key component of the AJB payment system, responsible for securely processing transactions between the point-of-sale terminals and the back-end financial systems. This check is critical for preventing interruptions in credit, debit, and other payment transactions at the store and gas pumps.

PROCS OK: 3 processes with PPID = 13670 | procs=3;;1:1024;0,

300

PROCS CRITICAL: PID file not found

AJB RTS SAF

ajb_rts_saf

Monitors the AJB Retail Transaction Switch (RTS) Store and Forward (SAF) functionality. When a connection to the payment host is unavailable, transactions are stored locally as record files (rec files) in the /fipay/configfiles/ directory. This check validates that these files are being processed and cleared in a timely manner. An alert for an old file indicates a failure in the forwarding process, which could cause a backlog of unsent transactions and impact daily sales reconciliation.

OK

120

WARN - 100 rec file(s)

CRITICAL - 100 rec file(s) older than 100 seconds

Binfile Age

binfile_age

This check monitors the age of the local BIN (Bank Identification Number) file to ensure it is kept up-to-date. This file contains critical routing information for credit and debit card transactions. An outdated BIN file can lead to payment failures, incorrect transaction routing, or a non-compliant payment system. The check will alert if the file's age exceeds the configured threshold, which has a default value of 7 days.

OK: File /fipay/binfile not found or newer than 7 days

3600

Warning: File /fipay/binfile is older than 7 days

Dccbinfile Age

dccbinfile_age

This check monitors the age of the Dynamic Currency Conversion (DCC) BIN file to ensure it contains up-to-date card information for international transactions. This file is essential for the system to correctly identify which credit cards are eligible for DCC, allowing customers to pay in their home currency. An outdated file can cause DCC transactions to fail or be improperly processed. The check will alert if the file's age exceeds the configured threshold, which has a default value of 7 days.

OK: File /fipay/dccbinfile not found or newer than 7 days

3600

Warning: File /fipay/dccbinfile is older than 7 days

SC//Reliant Platform Manager Events

rpm-events

This check monitors for the presence of keepalive signals, which are periodic "heartbeat" messages sent to SC//Reliant Platform Manager. The check's purpose is to ensure that the monitored system is alive and communicating. A failure to receive these messages within the interval indicates a potential issue with the system's client-side monitoring agent or network connectivity, suggesting a host may be offline or unreachable.

Note: Because this check depends on timely heartbeats, clock drift between a node and the Sensu Go servers can produce false rpm-events keepalive alerts. Keep Network Time Protocol (NTP) time synchronization running on every node so that nodes and servers agree on the clock.

Active Nodes: {"sit0000red0"=>2, "sit0001red0"=>2, "sit0002srv0"=>0} SC//Reliant Platform Manager response code: 201 SC//Reliant Platform Manager response message: Created

60

Active Nodes: {"sit0000red0"=>2, "sit0001red0"=>2, "sit0002srv0"=>0} Missing SECRET_KEY or AUTH_TOKEN

HTTPS Server

https_server

This check validates the health and availability of an HTTPS service by attempting to establish a secure connection on port 443. It ensures the service is not only running, but is also correctly handling SSL/TLS traffic. A failed check indicates that the HTTPS service is either down, misconfigured, or that its SSL certificate has issues, preventing secure communication.

OK - $http_version $http_code $http_message

900

CRITICAL - Timeout, unable to connect after $timeout seconds

CRITICAL - Remote endpoint is not TLS/SSL capable: $error_details

CRITICAL - Unable to solve $hostname hostname

OpenVPN Server

openvpn_server

This check verifies the operational status of an OpenVPN server instance. Its primary function is to confirm the service is running, while also reporting the number of currently connected clients. This provides a quick and clear view of both the service's health and its current utilization. A failed check could indicate the OpenVPN service is down, or that there's a problem with client connections.

OK- Instance status: - 1194 is up (1 clients connected) 1 clients connected on all instances

OK- Instance status: - 1194 is up (1 clients connected) - 1195 is up (1 clients connected) 2 clients connected on all instances

900

CRITICAL - Instance status: - 1194 is DOWN

CRITICAL - Instance status: - 1194 is up (1 clients connected) - 1195 is DOWN 1 clients connected on all instances Instance fault tolerance (40%) reached!

CRITICAL - Instance status: - 1194 is up (3 clients connected) * Maximum connected clients limit of 3 reached - 1195 is up (0 clients connected) 3 clients connected on all instances

Cluster Maintenance

cluster_maintenance

This check monitors the operational state of the cluster to determine if it is in a planned maintenance or standby mode. When a cluster is in maintenance mode, Pacemaker and Corosync will stop managing resources, and automatic failovers will be disabled. Similarly, if a specific node is in standby mode, it will not host any resources. This check is crucial for preventing false alerts on inactive nodes or services and for ensuring the cluster's high availability features are operational when they are supposed to be.

OK

600

CRITICAL: maintenance-mode enabled

CRITICAL: node $(hostname) is in standby

CRITICAL: maintenance-mode enabled and node $(hostname) in standby

CRITICAL: [Primary: DOWN] [Secondary: UP]

Filesystem Writable

fs_writable

This check verifies that monitored filesystems are writable. A critical alert is issued if any supported filesystem is detected as unwritable. The supported filesystem types are ext4, vfat, and nfs4.

Filesystem OK

OK - / Filesystem is writable

OK - /docker/shared-main_shared_volume Filesystem is writable

300

Filesystem CRITICAL

CRIT - /mnt READ ONLY FILESYSTEM

DNS (bind9)

bind9

This check verifies the health of the BIND9 DNS service. Its primary function is to confirm the bind9 daemon is actively running. An optional target can be configured to perform a DNS lookup, which validates that the service is not only running but also correctly resolving hostnames. A failed check may indicate the service is down, misconfigured, or that a DNS zone file issue is preventing successful resolution.

OK: DNS (bind) service is running

OK: DNS (bind9) is running and name resolution for ${TARGET} is resolving

300

CRITICAL: DNS (bind9) service is not running

CRITICAL: DNS (bind9) service is running but name resolution for ${TARGET} is not resolving

Service check reference (Windows hosts)

Service Check Name Description Status Examples Default Interval (seconds)

Windows Disk Utilization

windows_disk_check

Monitors the amount of used disk space on a Windows file system.

CheckDisk OK: All disk usage under XX%

60

CheckDisk WARNING: 1 disks in warning state (C:) XX.XX%, FREE: Y GB, SIZE: ZZ GB

CheckDisk CRITICAL: 1 disks in critical state (C:) XX.XX%, FREE: Y GB, SIZE: ZZ GB

Windows CPU Load Utilization

windows_cpu

Monitors the CPU load of a Windows host.

CheckCPU OK - value = XX.XX | system_cpu=XX.XX

60

CheckCPU WARNING - value = XX.XX | system_cpu=XX.XX

CheckCPU CRITICAL - value = XX.XX | system_cpu=XX.XX

Windows Memory Utilization

windows_mem_check

Monitors the percentage of Windows system memory currently in use.

CheckWindowsRAMLoad OK: RAM at XX%.

60

CheckWindowsRAMLoad WARNING: RAM at XX%

CheckWindowsRAMLoad CRITICAL: RAM at XX%

Windows Alohakitchen Process

windows_alohakitchen_check

This check monitors the alohakitchen process to verify that it is running as expected. The check will alert if the process is not found, indicating a potential service interruption.

OK : AlohaKitchen running.

120

CRITICAL: AlohaKitchen not found!

Windows Aloha Iber Process

windows_iber_check

This check verifies that the iber process is running on the system. The iber process is the core Front-of-House (FOH) application for the Aloha Point-of-Sale (POS) system. Its status is critical for ensuring that order entry and other key terminal functions are operational. This check will trigger an alert if the process is not found, indicating a potential disruption to the POS system.

OK : Iber running.

120

CRITICAL: Iber not found!

Windows Hospitality Alohatogo Process

windows_hospitality_alohatogo_check

This check verifies that the Radiant.Hospitality.AlohaToGo.ServiceHost process is running. This process is a critical background service for the Aloha ToGo online ordering system. Its active status is essential for processing online orders and ensuring the integration between the online platform and the point-of-sale system. The check will alert if the process is not running, which would indicate a potential failure in the online ordering functionality.

OK : Radiant.Hospitality.AlohaToGo.ServiceHost running

120

CRITICAL: Radiant.Hospitality.AlohaToGo.ServiceHost not found!

Aloha EdcSvr Service Process

windows_edcsvr_check

This check monitors the EdcSvr process to confirm it is running. The EdcSvr is the Electronic Data Capture service for the Aloha Point-of-Sale system and is critical for processing all electronic payments. The check will alert if the process is not active, which would indicate a service failure that could prevent payment card transactions.

OK : EdcSvr running.

120

CRITICAL: EdcSvr not found!

Aloha CtlSvr Service Process

windows_ctlsvr_check

This check verifies that the CtlSvr process is running. As the central Control Server for the Aloha Point-of-Sale system, this process is responsible for managing the system's core functions, including communication with POS terminals and system-wide data updates. An inactive CtlSvr process would indicate a critical failure in system management.

OK : CtlSvr running.

120

CRITICAL: CtlSvr not found!

Aloha OLOAlohaService Service Process

windows_oloalohaservice_check

This check verifies that the OLOAlohaService process is running. This service is a key component for the online ordering (OLO) integration with the Aloha Point-of-Sale system. An active status is essential for ensuring that online orders are properly received and processed by the in-store system. The check will alert if the process is not running, which would indicate a failure in the online ordering functionality.

OK : OLOAlohaService running.

120

CRITICAL: OLOAlohaService not found!

Aloha HttpsProxyService Service Process

windows_httpsproxyservice_check

This check monitors the HttpsProxyService to ensure it is running. This service is a key component for handling secure communications within the Aloha Point-of-Sale system, particularly for credit card processing (EMV) and other integrations that require secure data transfer. The check will alert if the process is not running, which could indicate a failure in secure payment and data functionality.

OK: HttpsProxyService running.

120

CRITICAL: HttpsProxyService not found!

Transactions File Size

windows_offxxxx_file_check

This check monitors for the presence of offline electronic funds transfer (EFT) files, typically named offXXXX, which indicate that transactions were unable to be processed in real-time. These files are created when a server loses network connectivity to the payment processor. The presence of these files is a critical indicator that payments are being held in a local queue and have not been finalized. The check will alert if these files are detected, signaling that the system is operating in offline mode and that action may be required to resolve the network issue and transmit the queued transactions.

OK: CheckWindowsFileSize OK: File size is equal or less than $WARNING.

120

Warning: CheckWindowsFileSize WARNING: File size is greater than $WARNING.

Critical: CheckWindowsFileSize CRITICAL: File $FILEPATH does not exist

Critical: CheckWindowsFileSize CRITICAL: File size is greater than $CRITICAL.