Linux Event Source Example

This section shows sample input, filters and output configuration to collect system and audit events from CentOS.

Input Plugin

An input plugin enables a specific source of events to be read by Logstash. The following code represents an example input plugin.

Copy

input-beats.conf

# Below input block collects events using beats plugins (e.g filebeats, auditbeats)
# Skip this block if it's already defined in another pipeline.
input {
    beats {
        port => 5044
    }
}

Make sure that port 5044 is open on the Logstash machine. As an example, if Logstash is on a CentOS system, run the following commands to open port 5044:

firewall-cmd --add-port=5044/tcp
firewall-cmd --add-port=5044/tcp --permanent
firewall-cmd --reload

Output Plugin

An output plugin sends event data to a particular destination. Outputs are the final stage in the event pipeline.

Copy

output-netwitness-tcp.conf

# Below is tcp output plugin with netwitness codec to tranform events in syslog and send it to LogDecoder
# Only one of these configurations can be within the same pipeline.
output {
  #if [@metadata][nw_type] { # Only targeted NetWitness items
    tcp {
      id => "netwitness-tcp-output-conf-output"
      host => "10.10.100.100"  ## LogDecoder IP
      port => 514
      ssl_enable => false
      #ssl_verify => true
      #ssl_cacert => "/path/to/certs/nw-truststore.pem"
      #ssl_key => "/path/to/certs/privkey.pem"
      #ssl_cert => "/path/to/certs/cert.pem"

      codec => netwitness {
        # Payload format mapping by nw_type.
        # If nw_type is absent or formatting fails, JSON event is used as the payload
        payload_format => {
          "apache" => "%APACHE-4-%{verb}: %{message}"
        }
        # Failover format, if above format fails
        # If nw_type is absent or formatting fails, JSON event is used as the payload
        payload_format_failover => {
          "apache" => "%APACHE-4: %{message}"  # When verb is missing
        }
      }
    }
  #}
}

Filter Plugin

A filter plugin performs intermediary processing on an event. Below is a filter plugin configuration for system events collected from linux using the Filebeat plugin.

Copy

linux-system.conf

# Filters are often applied conditionally depending on the characteristics of the events.
# Requires these additional configurations within the same pipeline: 
#   input-beats.conf
#   output-netwitness-tcp.conf

filter {
if ![@metadata][nw_type] {
  if [ecs][version] and [host][hostname] and [agent][type] == "filebeat" {
    if [event][module] == "system" {
      mutate {
        add_field => {
          "[@metadata][nw_type]" => "linux"
          "[@metadata][nw_msgid]" => "LOGSTASH001"
          "[@metadata][nw_source_host]" => "%{[host][hostname]}"
        }
      }
    } 
  }
 }
}

Below is filter plugin configuration for audit events collected from linux using the Auditbeat plugin.

Copy

linux-audit.conf

filter {
    if ![@metadata][nw_type] { # Update Once
      if [ecs][version] and [host][hostname] and [agent][type] == "auditbeat" {
        if [event][module] == "audit" {
          mutate {
            add_field => {
              "[@metadata][nw_type]" => "linux"
              "[@metadata][nw_msgid]" => "LOGSTASH002"
              "[@metadata][nw_source_host]" => "%{[host][hostname]}"
            }
          }
        }  
      }
    }
  }

Create a Pipeline

It is recommended to have one pipeline for each input type. For example, all beats collection should be in the same pipeline. To run collection as separate pipeline, create a directory and add above input, filters, and output configuration files to it.

Copy

Example Pipeline for Beats

 /etc/logstash/pipeline1/
 /etc/logstash/pipeline1/input-beats.conf
 /etc/logstash/pipeline1/output-netwitness-tcp.conf
 /etc/logstash/pipeline1/linux-system.conf
 /etc/logstash/pipeline1/linux-audit.conf

Modify /etc/logstash/pipeline.yml and add the following entries:

Copy

Add to pipeline.yml

  - pipeline.id: my-sample-pipeline-1
    path.config: "/etc/logstash/pipeline1/*.conf"