Log filters

Log filters are expression-based conditions that allow log messages to be filtered before they are forwarded to external services. With filters, only the messages that match the defined conditions are forwarded to the destination server, reducing noise downstream and limiting bandwidth used by log forwarding.

Log filters are available only in the context of the rsyslog client (profiles::rsyslog_client). For the broader configuration model, see Configuring log forwarding.

How filters are defined and applied

Filters are defined in the Rsyslog Client hash under the Filters property. Defining a filter on its own does not change which messages are forwarded; filters must be applied on a per-server basis. To use a filter, specify it in the server definition (Rsyslog server, Elasticsearch, Datadog, or New Relic).

When multiple filters are specified on a single server, those filters are linked with an OR operator: a log message that passes through at least one of the specified filters is forwarded to the server.

Inside a single filter, the Or and And blocks combine with the following logic:

• Inside the Or block, the listed conditions are linked with the or operator.
• Inside the And block, the listed conditions are linked with the and operator.
• Between the Or block and the And block, an and operator is used: the message must satisfy the And expression AND any of the Or expressions.

Between multiple filters applied to the same server, the expressions of each filter are linked with the or operator.

Filter definition options

The definition of a filter has the following options:

---
Rsyslog Client:
  Filters:
    <Filter Name>:
      Or:
        Contains:
          - <keyword>
        Not Contains:
          - <keyword>
        Regex:
          - <regex>
        Program Name:
          - <programname>
        Not Program Name:
          - <programname>
      And:
        Contains:
          - <keyword>
        Not Contains:
          - <keyword>
        Regex:
          - <regex>
        Program Name:
          - <programname>
        Not Program Name:
          - <programname>

Each condition inside Or or And accepts a list of values and has the following meaning:

• Contains. The log message must contain the given keyword.
• Not Contains. The log message must not contain the given keyword.
• Regex. The log message must match the given regular expression.
• Program Name. The programname field must exactly match the given value.
• Not Program Name. The programname field must not match the given value.

Considerations

• Filters are case sensitive. When defining filters, the words Hello, hElLo, HELLO, and heLLo are all considered different.
• Whitespace is significant. Leading and trailing whitespace characters are considered part of the expression and are not stripped from input data.
• Performance. Using too many filters can impact performance on the node running the rsyslog client. Define filters narrowly and reuse them across servers where possible.

Example: combined And/Or filter

Consider the following filter:

---
Rsyslog Client:
  Filters:
    'filter_name':
      Or:
        Contains:
          - a
          - b
        Not Contains:
          - c
        Regex:
          - '*a*regex'
      And:
        Contains:
          - d
        Not Contains:
          - g
        Regex:
          - 'another*'
          - '*regex*'

When applied to a server, this filter generates the following expression:

if (( $msg contains('d') and not($msg contains('g')) and re_match($msg, 'another*') and re_match($msg, '*regex*')) and ($msg contains('a') or $msg contains('b') or not($msg contains('c')) or re_match($msg, '*a*regex')))

The generated expression has an and condition between the And block and the Or block. The log message must satisfy the And expression and any of the Or expressions.

The log message another d regex passes this filter because:

• It contains d.
• It does not contain g.
• It matches both the another* and *regex* regular expressions.
• The Or block also accepts it because it contains a.

Filter scenarios

The following table shows common filter configurations and the rsyslog expressions they generate.

---
Rsyslog Client:
  Filters:
    'filter_1':
      Or:
        Contains:
          - Hello
          - World

---
Rsyslog Client:
  Filters:
    'filter_1':
      Or:
        Contains:
          - Hello
        Not Contains:
          - World

---
Rsyslog Client:
  Filters:
    'filter_1':
      Or:
        Contains:
          - Hello
        Not Contains:
          - World
        Regex:
          - st*s*w*s

---
Rsyslog Client:
  Filters:
    'filter_1':
      And:
        Contains:
          - Hello
          - World

---
Rsyslog Client:
  Filters:
    'filter_1':
      And:
        Contains:
          - Hello
        Not Contains:
          - World

---
Rsyslog Client:
  Filters:
    'filter_1':
      And:
        Contains:
          - Hello
        Not Contains:
          - World
        Regex:
          - st*t*k

---
Rsyslog Client:
  Filters:
    'filter_1':
      And:
        Contains:
          - Hello
        Not Contains:
          - World
    'filter_2':
      And:
        Contains:
          - hey
          - there

---
Rsyslog Client:
  Filters:
    'special_containers':
      Or:
        Program Name:
          - docker.rabbitmq
          - docker.redis

---
Rsyslog Client:
  Filters:
    'special_containers':
      And:
        Not Program Name:
          - docker.cosign-test
          - docker.nginx

Related documentation

• Managing logs
• Configuring log forwarding
• Configuring local log storage