Reading logs on a node

This page covers reading logs on an SC//Reliant™ platform node: where the local logs live, how to read and filter them (including rotated, compressed files), how to read a log line, and how to handle the rsyslog spool when it fills a disk. It complements the configuration pages, which describe how logs are forwarded and stored. Reading logs is a read-only, safe activity and is usually the first step in diagnosing an issue.

Where logs live on a node

Each node logs locally through rsyslog and the systemd journal.

• /var/log/messages is the primary local log and the first place to look on a node. It aggregates messages from rsyslog, the systemd journal, Docker, and other sources.
• Rotated logs are older messages files that have been rotated and gzipped: /var/log/messages.1.gz, /var/log/messages.2.gz, and so on.
• The systemd journal is readable with journalctl (for example, journalctl -u <service> for one service, or journalctl -k for kernel messages).

On a node running the rsyslog server profile (profiles::rsyslog_server), forwarded logs from other hosts are stored per host under /var/log/hosts/<hostname>/messages, and the relay/forwarding stream is the file /var/log/hosts/messages. See Configuring local log storage for how that store is configured, including compression and retention.

Reading and filtering logs

Read current (uncompressed) logs directly, and read rotated .gz files without unpacking them.

# read the current primary log
less /var/log/messages
# read a rotated, gzipped log without unpacking it
zcat /var/log/messages.1.gz | less
vim /var/log/messages.1.gz          # vim reads gzipped files directly
# find one service's lines in a rotated log
zgrep puppet-agent /var/log/messages.1.gz
# filter to what matters, and drop noisy lines
grep <service> /var/log/messages
grep -v keepalive /var/log/messages

For reading the compressed host store on a log server (including a gztool example), see Reading compressed log files.

Anatomy of a log line

A syslog line has four parts. Knowing them lets you grep precisely, by host, by resource, or by time.

Jul  9 01:39:13 sit0001red0 puppet-agent[2708593]: Catalog compiled by sitcomp1.reliant

To filter by host and then by resource, chain grep:

grep <host> /var/log/hosts/messages | grep <service>

Consistent timestamps across nodes depend on time synchronization. Keep Network Time Protocol (NTP) running so that correlation across nodes and the log server is reliable.

When the rsyslog spool fills a disk

The rsyslog spool at /var/spool/rsyslog is a common cause of a filling filesystem. When rsyslog cannot forward fast enough (for example, the destination is unreachable), queued messages accumulate as queue1.0* files under the spool and can consume the disk, which may in turn trigger a disk_free alert.

# check overall disk usage first
sudo rvc sys storage
# inspect the rsyslog spool
ls -lh /var/spool/rsyslog/

Clear the spool per the approved runbook. Do not delete stored host logs under /var/log/hosts/ to reclaim space; that is the retained record. For the disk_free alert thresholds, see Monitoring check reference.

Retention, in brief

A node does not retain or index logs for the long term; your SIEM does. On-node files are rotated (rotation is size-based, roughly 50 MB by default) and compressed to messages.gz, and forwarding off the node gives durability that outlives any single node rebuild. Confirm the retention policy for your environment. Retention and compression behavior are configured through the rsyslog server profile; see Configuring local log storage.

Related documentation

• Managing logs: logging architecture, the client and server profiles, and SIEM integration.
• Configuring local log storage: the profiles::rsyslog_server profile, compression, and retention.
• Configuring log forwarding: the profiles::rsyslog_client profile and forwarding to SIEM destinations.