Anatomy of Sensu event data
Every check result is delivered as a Sensu event. This page describes the structure of that event data, so that you can filter and search it after it has been forwarded to a destination such as Elasticsearch. For an overview of how the platform uses Sensu, see Sensu (monitoring).
Every check result, keepalive, and metric travels through the pipeline as an event JSON object. This page describes the structure of that object so you can parse Sensu events when building handler integrations, Elasticsearch queries, or SIEM dashboards.
Top-level event structure
The event JSON contains several nested JSON objects:
{
"check": {}, // Sensu Go monitoring data
"metrics": {}, // Only present on events that carry metric data, such as Load Average, Disk, and RAM
"entity": {}, // Metadata related to the host where the event/message originated
"id": "", // The unique event ID
"metadata": {}, // Sensu Go metadata -- not applicable
"pipelines": null, // Pipeline resources assigned to the event; typically null on this platform
"tag": "sensu-event-logs",
"sequence": 0,
"timestamp": 0 // The timestamp of the event
}
| Element | Description |
|---|---|
check |
The Sensu Go monitoring data for the event, including the check result and the check's configuration. See The check element. |
metrics |
Metric data points. Only applicable to Sensu Go events that carry metric data, such as Load Average, Disk, and RAM. Each point has a name, value, and timestamp. |
entity |
Metadata related to the host where the event originated, including the hostname, OS and platform details, network interfaces and addresses, subscriptions, and the Sensu agent version. |
id |
The unique event ID. |
metadata |
Sensu Go metadata, such as the namespace. Not applicable for parsing actionable data. |
pipelines |
The Sensu Go pipeline resources assigned to process the event. Pipelines are named event-processing workflows made up of filters, mutators, and handlers, and are an alternative to assigning handlers directly to a check. The SC//Reliant platform routes events through the check's handlers attribute instead, so this field is typically null. For more information, see the Sensu Go pipelines documentation. |
tag |
An integration tag, for example sensu-event-logs. This field is dependent on the integration and the options set in SC//Reliant Platform Manager. |
timestamp |
The timestamp of the event, in Unix time. |
The check element
Each event's check element has further sub-elements. When parsing for actionable data, the primary sub-elements to filter on are listed below. Other items present in the element represent the check's configuration.
"check": {
"command": "/usr/lib/nagios/plugins/check_load -r -w {{ .annotations.cpu_utilization_warning | default 2}} -c {{ .annotations.cpu_utilization_critical | default 3}}", // Command issued by Sensu
"duration": 0.049736716, // Execution time of the check
"executed": 1687496425, // Time, in Unix time, the check was executed on the remote system
"history": [], // History for the check, including issued times
"issued": 1687496425, // Time issued when the check was executed
"output": "OK: All resources are up\n", // Output of the command
"state": "passing", // State of the check: passing/failed
"status": 0, // Status of the check: 0 for passing, 1 for warning, 2 for critical
"last_ok": 1687496425,
"occurrences": 20004,
"metadata": {}, // Check metadata, such as its name and additional annotations like a description
"processed_by": "<hostname>" // Agent node that processed the check
}
| Sub-element | Description |
|---|---|
command |
The command issued by Sensu, including any annotation-driven threshold defaults. |
duration |
The execution time of the check, in seconds. |
executed |
The time, in Unix time, the check was executed on the remote system. |
history |
The recent history for the check, an array of executed timestamps and their resulting status codes. |
issued |
The time the check execution was issued. |
output |
The output of the command, for example OK: All resources are up. For checks that emit metrics, the output also carries performance data in nagios_perfdata format. |
state |
The state of the check: passing or failed. |
status |
The status code of the check: 0 for passing, 1 for warning, 2 for critical. |
last_ok |
The last time, in Unix time, the check returned a passing status. |
occurrences |
The number of consecutive events with the same status. |
metadata |
Check metadata, such as the check name (metadata.name) and annotations like a description. |
processed_by |
The agent node that processed the check. |
Example payload after Elasticsearch ingestion
The following is an example of a full cpu_utilization event payload after ingestion by Elasticsearch. The check.history array (one entry per execution) is truncated for readability. Note that ingestion adds an @timestamp, an event.ingested timestamp, and a flattened fields object that exposes every nested value as a dot-notation keyword field (for example, check.metadata.name.keyword or entity.system.hostname.keyword) for searching and aggregation.
{
"sequence": 0,
"metadata": {
"namespace": "default"
},
"@timestamp": "2023-07-31T07:31:38.859444-07:00",
"pipelines": null,
"tag": "sensu-event-logs",
"check": {
"occurrences": 49502,
"subscriptions": [
"physical",
"cpu_utilization"
],
"stdin": false,
"metadata": {
"name": "cpu_utilization",
"namespace": "default",
"annotations": {
"description": "Check load average"
},
"labels": {
"sensu.io/managed_by": "sensuctl"
}
},
"round_robin": false,
"proxy_entity_name": "",
"executed": 1690808649,
"is_silenced": false,
"timeout": 0,
"duration": 0.008199477,
"output": "OK - load average: 0.00, 0.00, 0.01|load1=0.000;2.000;3.000;0; load5=0.000;2.000;3.000;0; load15=0.005;2.000;3.000;0; \n",
"scheduler": "",
"occurrences_watermark": 49502,
"pipelines": [],
"last_ok": 1690808649,
"check_hooks": null,
"state": "passing",
"issued": 1690808648,
"high_flap_threshold": 0,
"subdue": null,
"total_state_change": 0,
"history": [
{
"executed": 1690808349,
"status": 0
},
{
"executed": 1690808649,
"status": 0
}
],
"ttl": 0,
"output_metric_format": "nagios_perfdata",
"secrets": null,
"command": "/usr/lib/nagios/plugins/check_load -r -w {{ .annotations.cpu_utilization_warning | default 2}} -c {{ .annotations.cpu_utilization_critical | default 3}}",
"env_vars": null,
"output_metric_handlers": null,
"handlers": [
"pagerduty"
],
"publish": true,
"interval": 300,
"low_flap_threshold": 0,
"runtime_assets": null,
"status": 0
},
"id": "95dd0033-be7e-4678-9c8b-7a8344331499",
"metrics": {
"handlers": null,
"points": [
{
"name": "load1",
"value": 0,
"timestamp": 1690808649,
"tags": null
},
{
"name": "load5",
"value": 0,
"timestamp": 1690808649,
"tags": null
},
{
"name": "load15",
"value": 0.005,
"timestamp": 1690808649,
"tags": null
}
]
},
"event": {
"ingested": "2023-07-31T14:42:55.297780682Z"
},
"entity": {
"subscriptions": [
"all",
"physical",
"cluster",
"entity:sit0007red1"
],
"deregistration": {},
"metadata": {
"name": "sit0007red1",
"namespace": "default"
},
"system": {
"vm_system": "kvm",
"vm_role": "host",
"hostname": "sit0007red1",
"processes": null,
"os": "linux",
"platform_version": "8.11",
"libc_type": "glibc",
"cloud_provider": "",
"arch": "amd64",
"platform_family": "debian",
"platform": "debian",
"network": {
"interfaces": [
{
"addresses": [
"127.0.0.1/8"
],
"name": "lo"
},
{
"addresses": [
"192.168.1.5/29"
],
"name": "eth0",
"mac": "7c:c2:55:11:09:ee"
},
{
"addresses": [
"10.194.145.69/27"
],
"name": "eth1",
"mac": "7c:c2:55:11:09:ef"
},
{
"addresses": null,
"name": "eth2",
"mac": "7c:c2:55:11:09:f0"
},
{
"addresses": [
"172.16.16.2/30"
],
"name": "eth3",
"mac": "7c:c2:55:11:09:f1"
},
{
"addresses": null,
"name": "eth3.202",
"mac": "7c:c2:55:11:09:f1"
},
{
"addresses": [
"192.168.1.12/29"
],
"name": "br0",
"mac": "7c:c2:55:11:09:f1"
}
]
}
},
"last_seen": 1685615414,
"redact": [
"password",
"passwd",
"pass",
"api_key",
"api_token",
"access_key",
"secret_key",
"private_key",
"secret"
],
"deregister": false,
"entity_class": "agent",
"user": "agent",
"sensu_agent_version": "(devel)"
},
"timestamp": 1690808649
}
Useful fields for filtering
When querying ingested events in Elasticsearch or a SIEM, the following flattened fields are the most useful starting points:
| Field | Example value | Use |
|---|---|---|
check.metadata.name.keyword |
cpu_utilization |
Filter events by check name. |
check.status |
0, 1, 2 |
Filter by severity: passing, warning, or critical. |
check.state.keyword |
passing |
Filter by passing/failed state. |
check.output.keyword |
OK - load average: ... |
Inspect the raw check output, including performance data. |
check.handlers.keyword |
pagerduty |
Identify which handler(s) the event was routed to. |
entity.system.hostname.keyword |
sit0007red1 |
Filter events by originating host. |
metrics.points.name / metrics.points.value |
load1 / 0.005 |
Graph or alert on metric values. |
Related documentation
- Configuring service checks: defining what Sensu monitors.
- Configuring handlers: routing events to destinations.
- Service check reference: built-in service-check catalog.