Anatomy of Sensu event data

Anatomy of Sensu event data

Every check result is delivered as a Sensu event. This page describes the structure of that event data, so that you can filter and search it after it has been forwarded to a destination such as Elasticsearch. For an overview of how the platform uses Sensu, see Sensu (monitoring).

Every check result, keepalive, and metric travels through the pipeline as an event JSON object. This page describes the structure of that object so you can parse Sensu events when building handler integrations, Elasticsearch queries, or SIEM dashboards.

Top-level event structure

The event JSON contains several nested JSON objects:

{
  "check": {},      // Sensu Go monitoring data
  "metrics": {},    // Only present on events that carry metric data, such as Load Average, Disk, and RAM
  "entity": {},     // Metadata related to the host where the event/message originated
  "id": "",         // The unique event ID
  "metadata": {},   // Sensu Go metadata -- not applicable
  "pipelines": null,  // Pipeline resources assigned to the event; typically null on this platform
  "tag": "sensu-event-logs",
  "sequence": 0,
  "timestamp": 0    // The timestamp of the event
}
Element Description
check The Sensu Go monitoring data for the event, including the check result and the check's configuration. See The check element.
metrics Metric data points. Only applicable to Sensu Go events that carry metric data, such as Load Average, Disk, and RAM. Each point has a name, value, and timestamp.
entity Metadata related to the host where the event originated, including the hostname, OS and platform details, network interfaces and addresses, subscriptions, and the Sensu agent version.
id The unique event ID.
metadata Sensu Go metadata, such as the namespace. Not applicable for parsing actionable data.
pipelines The Sensu Go pipeline resources assigned to process the event. Pipelines are named event-processing workflows made up of filters, mutators, and handlers, and are an alternative to assigning handlers directly to a check. The SC//Reliant platform routes events through the check's handlers attribute instead, so this field is typically null. For more information, see the Sensu Go pipelines documentation.
tag An integration tag, for example sensu-event-logs. This field is dependent on the integration and the options set in SC//Reliant Platform Manager.
timestamp The timestamp of the event, in Unix time.

The check element

Each event's check element has further sub-elements. When parsing for actionable data, the primary sub-elements to filter on are listed below. Other items present in the element represent the check's configuration.

"check": {
  "command": "/usr/lib/nagios/plugins/check_load -r -w {{ .annotations.cpu_utilization_warning | default 2}} -c {{ .annotations.cpu_utilization_critical | default 3}}",  // Command issued by Sensu
  "duration": 0.049736716,    // Execution time of the check
  "executed": 1687496425,     // Time, in Unix time, the check was executed on the remote system
  "history": [],              // History for the check, including issued times
  "issued": 1687496425,       // Time issued when the check was executed
  "output": "OK: All resources are up\n",  // Output of the command
  "state": "passing",         // State of the check: passing/failed
  "status": 0,                // Status of the check: 0 for passing, 1 for warning, 2 for critical
  "last_ok": 1687496425,
  "occurrences": 20004,
  "metadata": {},             // Check metadata, such as its name and additional annotations like a description
  "processed_by": "<hostname>"  // Agent node that processed the check
}
Sub-element Description
command The command issued by Sensu, including any annotation-driven threshold defaults.
duration The execution time of the check, in seconds.
executed The time, in Unix time, the check was executed on the remote system.
history The recent history for the check, an array of executed timestamps and their resulting status codes.
issued The time the check execution was issued.
output The output of the command, for example OK: All resources are up. For checks that emit metrics, the output also carries performance data in nagios_perfdata format.
state The state of the check: passing or failed.
status The status code of the check: 0 for passing, 1 for warning, 2 for critical.
last_ok The last time, in Unix time, the check returned a passing status.
occurrences The number of consecutive events with the same status.
metadata Check metadata, such as the check name (metadata.name) and annotations like a description.
processed_by The agent node that processed the check.

Example payload after Elasticsearch ingestion

The following is an example of a full cpu_utilization event payload after ingestion by Elasticsearch. The check.history array (one entry per execution) is truncated for readability. Note that ingestion adds an @timestamp, an event.ingested timestamp, and a flattened fields object that exposes every nested value as a dot-notation keyword field (for example, check.metadata.name.keyword or entity.system.hostname.keyword) for searching and aggregation.

{
  "sequence": 0,
  "metadata": {
    "namespace": "default"
  },
  "@timestamp": "2023-07-31T07:31:38.859444-07:00",
  "pipelines": null,
  "tag": "sensu-event-logs",
  "check": {
    "occurrences": 49502,
    "subscriptions": [
      "physical",
      "cpu_utilization"
    ],
    "stdin": false,
    "metadata": {
      "name": "cpu_utilization",
      "namespace": "default",
      "annotations": {
        "description": "Check load average"
      },
      "labels": {
        "sensu.io/managed_by": "sensuctl"
      }
    },
    "round_robin": false,
    "proxy_entity_name": "",
    "executed": 1690808649,
    "is_silenced": false,
    "timeout": 0,
    "duration": 0.008199477,
    "output": "OK - load average: 0.00, 0.00, 0.01|load1=0.000;2.000;3.000;0; load5=0.000;2.000;3.000;0; load15=0.005;2.000;3.000;0; \n",
    "scheduler": "",
    "occurrences_watermark": 49502,
    "pipelines": [],
    "last_ok": 1690808649,
    "check_hooks": null,
    "state": "passing",
    "issued": 1690808648,
    "high_flap_threshold": 0,
    "subdue": null,
    "total_state_change": 0,
    "history": [
      {
        "executed": 1690808349,
        "status": 0
      },
      {
        "executed": 1690808649,
        "status": 0
      }
    ],
    "ttl": 0,
    "output_metric_format": "nagios_perfdata",
    "secrets": null,
    "command": "/usr/lib/nagios/plugins/check_load -r -w {{ .annotations.cpu_utilization_warning | default 2}} -c {{ .annotations.cpu_utilization_critical | default 3}}",
    "env_vars": null,
    "output_metric_handlers": null,
    "handlers": [
      "pagerduty"
    ],
    "publish": true,
    "interval": 300,
    "low_flap_threshold": 0,
    "runtime_assets": null,
    "status": 0
  },
  "id": "95dd0033-be7e-4678-9c8b-7a8344331499",
  "metrics": {
    "handlers": null,
    "points": [
      {
        "name": "load1",
        "value": 0,
        "timestamp": 1690808649,
        "tags": null
      },
      {
        "name": "load5",
        "value": 0,
        "timestamp": 1690808649,
        "tags": null
      },
      {
        "name": "load15",
        "value": 0.005,
        "timestamp": 1690808649,
        "tags": null
      }
    ]
  },
  "event": {
    "ingested": "2023-07-31T14:42:55.297780682Z"
  },
  "entity": {
    "subscriptions": [
      "all",
      "physical",
      "cluster",
      "entity:sit0007red1"
    ],
    "deregistration": {},
    "metadata": {
      "name": "sit0007red1",
      "namespace": "default"
    },
    "system": {
      "vm_system": "kvm",
      "vm_role": "host",
      "hostname": "sit0007red1",
      "processes": null,
      "os": "linux",
      "platform_version": "8.11",
      "libc_type": "glibc",
      "cloud_provider": "",
      "arch": "amd64",
      "platform_family": "debian",
      "platform": "debian",
      "network": {
        "interfaces": [
          {
            "addresses": [
              "127.0.0.1/8"
            ],
            "name": "lo"
          },
          {
            "addresses": [
              "192.168.1.5/29"
            ],
            "name": "eth0",
            "mac": "7c:c2:55:11:09:ee"
          },
          {
            "addresses": [
              "10.194.145.69/27"
            ],
            "name": "eth1",
            "mac": "7c:c2:55:11:09:ef"
          },
          {
            "addresses": null,
            "name": "eth2",
            "mac": "7c:c2:55:11:09:f0"
          },
          {
            "addresses": [
              "172.16.16.2/30"
            ],
            "name": "eth3",
            "mac": "7c:c2:55:11:09:f1"
          },
          {
            "addresses": null,
            "name": "eth3.202",
            "mac": "7c:c2:55:11:09:f1"
          },
          {
            "addresses": [
              "192.168.1.12/29"
            ],
            "name": "br0",
            "mac": "7c:c2:55:11:09:f1"
          }
        ]
      }
    },
    "last_seen": 1685615414,
    "redact": [
      "password",
      "passwd",
      "pass",
      "api_key",
      "api_token",
      "access_key",
      "secret_key",
      "private_key",
      "secret"
    ],
    "deregister": false,
    "entity_class": "agent",
    "user": "agent",
    "sensu_agent_version": "(devel)"
  },
  "timestamp": 1690808649
}

Useful fields for filtering

When querying ingested events in Elasticsearch or a SIEM, the following flattened fields are the most useful starting points:

Field Example value Use
check.metadata.name.keyword cpu_utilization Filter events by check name.
check.status 0, 1, 2 Filter by severity: passing, warning, or critical.
check.state.keyword passing Filter by passing/failed state.
check.output.keyword OK - load average: ... Inspect the raw check output, including performance data.
check.handlers.keyword pagerduty Identify which handler(s) the event was routed to.
entity.system.hostname.keyword sit0007red1 Filter events by originating host.
metrics.points.name / metrics.points.value load1 / 0.005 Graph or alert on metric values.