Troubleshooting a node that shows as offline

In SC//Reliant™ Platform Manager, each node in the Nodes list has a status indicator, and the node's own page shows Online or Offline with the time of the last update. The indicator reflects whether Platform Manager is receiving keepalive messages from the node's Sensu agent. It doesn't reflect whether the node itself is running. A node can be running its workloads and reachable over SSH and still show as offline, because its Sensu agent isn't reporting.

This page describes what the status indicator means and how to find and fix the most common reasons a node's Sensu agent stops reporting.

Understanding the node status

The rpm-events service check alerts when keepalive messages stop arriving. See Monitoring check reference. To see how many nodes in your environment are online, offline, or unknown, use the API. See Retrieve the node status summary.

Before you begin

The checks on this page are run on the node, because RVC doesn't provide methods for service status, service logs, or name resolution. See Connecting to a node.

In a cluster, check both nodes. The two nodes share their node group's configuration, so a problem that affects one usually affects the other.

Step 1: Check that the Sensu agent is installed and running

systemctl is-active sensu-agent
systemctl is-enabled sensu-agent

A running agent returns active and enabled. If the agent isn't installed, the commands return output like the following:

inactive
Failed to get unit file state for sensu-agent.service: No such file or directory

The Sensu agent is installed by the profiles::sensugo::agent Puppet profile. If the agent isn't installed, check the node group's Puppet Profiles in Platform Manager and add profiles::sensugo::agent if it's missing. See Updating a node group. Then run Puppet on the node:

sudo rvc puppet run

The Puppet run installs the sensu-go-agent package, writes the agent configuration to /etc/sensu/agent.yml, and starts the sensu-agent service. Continue with Step 2 to confirm that the agent can connect.

Step 2: Check that the agent can connect to the Sensu servers

Read the agent's recent log entries:

sudo journalctl -u sensu-agent --since "10 min ago" --no-pager | tail -20

The agent tries each Sensu server in turn. The servers it uses are listed under backend-url in /etc/sensu/agent.yml, and come from the Sensu Go Common Configurations property. See Node group property reference.

Step 3: Fix name resolution for the Sensu servers

Check whether the node can resolve the Sensu server names from backend-url, and which DNS servers it uses:

getent hosts sitsrv1
cat /etc/resolv.conf

If getent returns nothing, the node can't resolve the name. Puppet writes /etc/resolv.conf from the Hostips Dns property, followed by the Puppet server's IP address. If a public DNS server such as 8.8.8.8 is listed first, it answers that internal SC//Reliant host names don't exist, and the node doesn't try the next server. For example:

domain reliant
search reliant rb.reliant
nameserver 8.8.8.8
nameserver 192.0.2.10

To fix this, set Hostips Dns on the node group to the DNS server that resolves SC//Reliant host names for your environment. In many environments this is the same address as the Hostip Puppet property. If you aren't sure which address to use, contact SC//Reliant Support. Then run Puppet on the node and check again:

sudo rvc puppet run
cat /etc/resolv.conf
getent hosts sitsrv1

The public DNS server no longer appears in /etc/resolv.conf, and getent returns the server's address.

Step 4: Restart the agent and confirm

Restart the agent so it reconnects immediately rather than waiting for its next retry:

sudo systemctl restart sensu-agent
sudo journalctl -u sensu-agent --since "1 min ago" --no-pager | grep -i connect

A successful connection is logged as follows:

"backend-url":"ws://sitsrv1:8081","component":"agent","level":"info","msg":"successfully connected"

Within a minute or two, the node's status indicator in Platform Manager turns green. Repeat Steps 1 through 4 on the other node in the cluster.

NOTE: Keepalive checks depend on accurate time. If a node's clock drifts from the Sensu servers' clocks, the node can show keepalive alerts even when its agent is connected. Keep NTP time synchronization running on every node.

Related documentation

• Monitoring your environment: how the SC//Reliant platform uses Sensu.
• Monitoring check reference: the rpm-events keepalive check and the other service checks.
• Retrieve the node status summary: counting online, offline, and unknown nodes through the API.
• Node group property reference: the Hostips Dns and Sensu Go properties.