Monitoring your environment

The SC//Reliant™ platform uses Sensu, an open-source monitoring event pipeline, to collect, process, and forward monitoring telemetry. A Sensu agent on each edge node collects monitoring events and forwards them to a cluster of Sensu servers in the SC//Reliant Management Plane. The Sensu servers can then forward the events to a third-party destination such as PagerDuty.

Overview

The SC//Reliant platform provides real-time monitoring across the fleet from a scalable server cluster: node health, resource utilization (CPU, RAM, disk, network I/O), workload status (containers and VMs), and per-site performance metrics. Drill-down diagnostics on any individual site or node are available from the unified view, so operators are one click away from the data they need when investigating an issue.

Integrated Sensu Go provides low-level, granular monitoring and alerting, consolidating telemetry into an event stream that can be directed to a customer SIEM or into the SC//AcuVigil™-provided Elastic instance. Sensu Go supports both metric-based monitoring and threshold-based alerting, and virtually any resource, including at the application level, can be reported on, supporting deep observability and telemetry.

Operations live in one place. SC//Reliant Platform Manager handles configuration, orchestration, and deployment. SC//AcuVigil delivers real-time health, network, and infrastructure monitoring. Together they cover every site, every node, and every workload from a unified web console.

Utility-grade operations model

The SC//Reliant platform delivers a utility-grade day-2 operations model through three layers working in concert:

SC//Reliant Platform Manager continuously monitors workload and node health, firing alerts on threshold breaches before downstream impact.
Puppet's declarative state model auto-remediates configuration drift, crashed services, and most operational deviations on every agent run (default every 30 minutes).
SC//Reliant Platform Manager webhooks escalate any unresolved condition to the managed-services tier and to ITSM platforms such as ServiceNow before on-site staff are aware anything happened.

In production, this means container crashes auto-restart per declared policy with no operator action and no site impact. Configuration drift auto-corrects on the next Puppet run. Failed package updates auto-roll back to the prior immutable image tag in under five minutes. Primary Wide Area Network (WAN) failure triggers automatic LTE/cellular failover within seconds while site workloads continue uninterrupted. And predictive disk health degradation fires a ticket and pre-stages parts before the disk fails.

Automated issue detection

SC//Reliant Platform Manager provides continuous health and anomaly monitoring across the full stack:

Servers. CPU, RAM, disk health via SMART, fans, and thermal via Intelligent Platform Management Interface (IPMI).
Network. Interface state, throughput, errors, and primary WAN reachability via the continuous monitoring script.
Workloads. Container and VM resource consumption, restart counts, and application metrics through Prometheus endpoints exposed by service containers.
Peripherals. POS terminals, payment devices, and other on-site controllers monitored via protocol adapter containers.

Threshold-based anomaly detection is native to the platform via rsyslog and Sensu. Advanced anomaly detection, statistical drift and ML-based, deploys against the customer SIEM, alongside any telemetry containers stood up at the site (Prometheus + Alertmanager, Splunk, Elastic, Datadog, Dynatrace) consuming locally generated telemetry data.

Alerting and notification

SC//Reliant Platform Manager provides built-in alerting with configurable thresholds tunable per metric, per site, per group, or per workload.

Native channels are email and webhook. Webhook integration drives downstream ticketing systems (ServiceNow, Jira, PagerDuty, Opsgenie) and notification platforms through standard REST. SMS and mobile push are typically delivered by routing webhook events through a notification service such as Twilio or the PagerDuty mobile app rather than as a built-in primitive.

Automated ticket creation

SC//Reliant Platform Manager webhooks integrate directly with the ServiceNow REST API for automatic incident creation. Configurable elements include:

Trigger conditions: which alert classes create tickets versus notify only.
Severity mapping from SC//Reliant Platform Manager severity to ServiceNow priority.
Routing rules: assignment group based on site, brand, asset class, or condition type.
Classification, category, subcategory, and CI lookup against the ServiceNow Configuration Management Database (CMDB) by serial number or hostname.
Payload templates with custom fields populated automatically, site ID, brand, device, condition, recent log excerpts, recommended remediation, and a runbook link.

All settings are configurable per metric per brand, and the classification taxonomy is aligned to the customer ServiceNow incident management model during deployment. Bidirectional updates are supported: ticket state changes in ServiceNow can fire webhooks back to SC//Reliant Platform Manager for status reconciliation.

Predictive failure detection

The SC//Reliant platform can ingest SMART data from drives, Baseboard Management Controller (BMC) sensor data from motherboards (thermal, fan, voltage), and workload trend metrics for predictive failure analysis.

Data is forwarded into the customer SIEM with predictive signals tracked, including:

SMART threshold trending for disks (write errors, reallocated sectors trending, pending sector counts).
Thermal trending: sustained high-temperature events predict cooling failure.
Memory error trending, correctable ECC errors trending up predict imminent uncorrectable failure.
Storage capacity trending, predicting near-full conditions before service impact.
Fan RPM trending, predicting fan failure.

Maintenance is scheduled via a SIEM webhook into ServiceNow change management. Replacement parts are ordered before failure, and a technician is dispatched during the scheduled change window with the right part in hand. This reduces unscheduled outages and moves operations away from reactive truck-rolls.

Proactive device intelligence

The SC//Reliant platform correlates failure events from inventoried devices through the central event and log stream, rsyslog forwarding into the customer SIEM (Elastic, Splunk, or other) in current production deployments.

Pattern detection identifies recurring failure clusters across hardware models, firmware versions, serial number ranges (suggesting manufacturer batch issues), brand cohorts, and geographic clusters. Detected patterns surface as proactive alerts in SC//Reliant Platform Manager and as ServiceNow problem records. Variablized data can include fields such as endpoint serial numbers, models, and other site-specific data points. This information is folded into the event and log stream forwarded into the SIEM.

For heterogeneous fleets spanning multiple hardware generations across thousands of sites, this proactive pattern detection is particularly valuable for catching common-mode failures before they cascade across multiple sites.

Automated remediation actions

The SC//Reliant platform performs a range of autonomous remediation actions without human intervention:

Container restart when a container is detected to be stopped or in a failed state.
VM restart depending on the nature of the VM and the type of crash.
Conditional restart via SC//Reliant Job Engine (RJE) jobs and script-based services that monitor for specific conditions, stale files, unavailable or closed network ports, and force restart of containers or VMs per declared restart policy with crash-loop limits.
Service recovery via Puppet re-convergence on the next agent run.
Dependency retry with exponential backoff when a workload fails because a dependency is unavailable.
Configuration drift correction on every Puppet run (file integrity, package state, network config, firewall rules).
Failed update rollback to the prior immutable container image tag or VM snapshot.
Workload migration to a surviving node in an HA cluster on hardware failure.
Primary WAN failover to LTE/cellular and reverse on restoration.
Disk cleanup via scheduled container jobs (log rotation, temp file purge, container image pruning).
Process kill via cgroup OOM enforcement when a workload exceeds quota.
Scheduled maintenance tasks (database vacuum, cache flush, certificate rotation).
Self-test execution on schedule.

Guardrails are central to the model: declared state in Puppet defines what is allowed, restart limits prevent crash loops from masking deeper issues, and alerts fire when remediation thresholds are exceeded so a human can investigate.

Automated software rollback

The SC//Reliant platform supports automated detection and rollback for failed updates. Post-update Puppet runs verify desired state, and SC//Reliant Platform Manager monitors workload health metrics against pre-update baselines, including CPU spike, memory exhaustion, restart loops, error log frequency, and response latency.

On regression detection, the platform runs one of the following, depending on the workload:

An automated rollback to the prior immutable container image tag. This typically completes in under five minutes because the prior image is cached on the node.
A revert of the Puppet configuration commit.
A revert of the VM snapshot, for VM-based workloads.

Rollback policies are defined in SC//Reliant Platform Manager. For example, auto-rollback on N consecutive failures, or manual approval required for production-critical workloads. Combined with staged rollout (pilot sites first, then groups, then fleet-wide), regressions are caught before fleet-wide impact.

Predictive capacity management

SC//Reliant Platform Manager collects per-node and per-workload resource utilization telemetry continuously across the fleet, covering CPU, memory, disk, network, and GPU when present. Threshold-based alerting on capacity trends is supported natively and is configurable per metric, per site, and per brand.

Forecasting and proactive right-sizing or rebalancing are typically delivered through customer-chosen analytics or AIOps tooling such as Splunk ITSI, Elastic, Datadog, Dynatrace, or Grafana Mimir, consuming SC//Reliant Platform Manager telemetry. A containerized capacity-planning service can also be deployed on the SC//Reliant platform for site-local forecasting.

Resources (CPU, memory, disk) can be assigned to workloads, both containers and VMs, on a not-to-exceed basis. Through the SC//Reliant Job Engine (RJE) management system, resource allocations can be changed on a scheduled or triggered basis. This is useful when a known event, such as overnight back-of-house reporting, on-site training, or a predicted peak window, calls for a different allocation than the steady state.

Remote diagnostics

Remote diagnostics from the cloud control plane include:

PCI-compliant encrypted remote shell access.
Real-time log streaming from any node.
Container and VM lifecycle control (start, stop, restart, redeploy).
Configuration push and revert.
Remote package install and update.
Network diagnostics (ping, traceroute, packet capture).
File system inspection.

The SC//Reliant platform also leverages out-of-band hardware management (IPMI, iLO, XClarity, iDRAC, vPro/BMC) for fencing and cluster management, which makes the same service set available for remote support, troubleshooting, and system recovery. Cellular WAN failover ensures remote access stays available even during a primary WAN outage.

The following documentation goes into the monitoring solution in more detail.

For the record of changes made in SC//Reliant Platform Manager, see Using the audit trail.