Configuring log forwarding

Log forwarding on the SC//Reliant™ platform is configured through the profiles::rsyslog_client Puppet profile. This profile configures a node to forward log messages to one or more destination servers and does not store any log messages locally. For local log storage, see Configuring local log storage.

The rsyslog client supports forwarding to the following destination types:

• Rsyslog servers
• Elasticsearch
• Datadog
• New Relic

The rsyslog client and rsyslog server profiles can be applied to the same node. When both are used together, the profiles::rsyslog_client profile must be ordered before profiles::rsyslog_server in the Puppet profiles list.

The Rsyslog Client hash

The profiles::rsyslog_client profile uses the Rsyslog Client (my_rsyslog_client) hash property to gather configuration data. The hash has the following structure:

---
Rsyslog Client:
  General Configurations:
    Version: <package version>
    Max Message Size: <maximum message size>
    Cleanup: <true|false>
  Filters:
    <Filter Name>:
      # See the Log filters page for the full definition
  Files:
    <Filename>:
      Path: <full file path>
      File Tag: <tag>
      Facility: <syslog facility>
      JSON-formatted File: <true|false>
      Destination Servers:
        Rsyslog:
          Servers:
            - <server_name>
        ElasticSearch:
          <server>:
            Elasticsearch Index: <index>
        Datadog:
          <server>:
            Source: <source>
        New Relic:
          Servers:
            - <server_name>
  Rsyslog Servers:
    Servers:
      <server_name>:
        Host: <hostname_or_ip>
        Port: <port>
        Protocol: <protocol>
        Filters:
          - <filter_name>
  Elasticsearch Servers:
    <server>:
      Elasticsearch URL: <url>
      Elasticsearch Index: <index>
      Port: <port>
      Authentication:
        Username: <username>
        Password: <password>
      Exclusive: <true|false>
      Filters:
        - <filter_name>
      Queue Parameters:
        Size: <queue size>
        Dequeue Batch Size: <batch size>
        Minimum Dequeue Batch Size: <minimum batch size>
  Datadog Servers:
    <server>:
      API Key: <api_key>
      Source: <source>
      Tags: <tags>
      Exclusive: <true|false>
      Filters:
        - <filter_name>
      Queue Parameters:
        Size: <queue size>
        Dequeue Batch Size: <batch size>
        Minimum Dequeue Batch Size: <minimum batch size>
  New Relic Servers:
    <server>:
      API Key: <api_key>
      Exclusive: <true|false>
      Filters:
        - <filter_name>
      Queue Parameters:
        Size: <queue size>
        Dequeue Batch Size: <batch size>
        Minimum Dequeue Batch Size: <minimum batch size>

The sections that follow describe each top-level field in detail.

General Configurations

The General Configurations block sets options that apply to the rsyslog client as a whole.

Filters

The Filters field defines log filters available for use across destination servers. Defining a filter here does not change which messages are forwarded; filters must be referenced from a server definition to take effect.

For the complete filter definition syntax (Or/And blocks, supported conditions, considerations, and worked examples), see Log filters.

Files

The Files field configures log files to be streamed to external services. If an application logs to a file instead of syslog or the journal, this feature forwards those log entries to a destination server of your choice.

Each file is configured independently. The same log message can be forwarded to multiple destination servers (Rsyslog, Elasticsearch, Datadog, or New Relic). The server name in the file definition is a reference to the matching server declaration in the Rsyslog Servers, Elasticsearch Servers, Datadog Servers, or New Relic Servers sections.

If no destination server is set for a file, the file content is sent to all servers that are not declared as Exclusive.

Required and optional fields per file

When forwarding a JSON-formatted log file to Elasticsearch, enable JSON-formatted File. With this field enabled, the data is parsed as JSON and renders cleanly in Elasticsearch Kibana. Datadog and New Relic parse JSON automatically as part of their log ingestion pipelines, so this field is not required for those destinations.

When the destination is an Elasticsearch or Datadog server, you can override the Elasticsearch Index or Datadog Source for the file. You may also reuse the same Index or Source that the server itself uses.

Example: forwarding a single JSON file to Elasticsearch

---
Rsyslog Client:
  Files:
    sensugo_events:
      Path: "/var/log/sensu/events.log"
      File Tag: sensugo_events
      JSON-formatted File: true
      Destination Servers:
        Elasticsearch:
          elastic-production:
            Elasticsearch Index: sensu-event-logs
  Elasticsearch Servers:
    elastic-production:
      Elasticsearch URL: https://example.es.aws.found.io
      Elasticsearch Index: sensu-event-logs
      Port: 443
      Authentication:
        Username: rpm_logs
        Password: <password>
      Exclusive: true

Example: forwarding two files to different destination servers

---
Rsyslog Client:
  Files:
    sensugo_events:
      Path: "/var/log/sensu/events.log"
      File Tag: sensugo_events
      JSON-formatted File: true
      Destination Servers:
        Elasticsearch:
          elastic-production:
            Elasticsearch Index: sensu-event-logs
    httpd_access:
      Path: "/var/log/apache2/access.log"
      File Tag: apache
      JSON-formatted File: false
      Destination Servers:
        Datadog:
          customer-datadog:
            Source: apache
  Elasticsearch Servers:
    elastic-production:
      Elasticsearch URL: https://example.es.aws.found.io
      Elasticsearch Index: sensu-event-logs
      Port: 443
      Authentication:
        Username: user
        Password: <password>
      Exclusive: true
  Datadog Servers:
    customer-datadog:
      API Key: <api_key>
      Source: files
      Exclusive: true

Rsyslog servers

The Rsyslog Servers field declares rsyslog servers that should receive log messages from the rsyslog client.

A declared rsyslog server receives log messages from the following sources:

• All system logs.
• Files that specify the rsyslog server as a Destination server.
• Files that do not specify any Destination server.

Third-party integrations

The rsyslog client also forwards logs to three third-party monitoring services: Elasticsearch, Datadog, and New Relic. Each is declared in its own block of the Rsyslog Client hash.

For each third-party server, the log sources are:

• All system logs.
• Files that specify the third-party server as a Destination server.
• Files that do not specify any Destination server.

If a third-party server is declared as Exclusive, the only source of logs is files that explicitly declare that server as a Destination server. This allows tightly scoping a destination to a specific file stream.

Elasticsearch

To forward logs to Elasticsearch, the rsyslog client needs the URL of the Elasticsearch server. If authentication is required, supply a username and password. An Elasticsearch Index can also be specified. If it is omitted, the default index logstash-rsyslog is used.

Datadog

To forward logs to Datadog, the rsyslog client needs a valid Datadog API key for the account that will receive the logs. Datadog also supports sending Source and Tags as metadata along with each log message.

New Relic

To forward logs to New Relic, the rsyslog client needs a valid New Relic API key for the account that will receive the logs.

Queue parameters

When the rsyslog client forwards logs to third-party servers (Elasticsearch, Datadog, New Relic), default queue parameters are applied to manage throughput. The defaults are tuned for both low and high throughput and typically do not need to be changed.

Related documentation

• Managing logs
• Configuring local log storage
• Log filters