What is LogStash:
LogStash is an Elastic product that can collect, parse, and transform logs to be presented to some type of output such as an Elastic Stack or a RSA Decoder or Virtual Log Collector.
https://www.elastic.co/downloads/logstash-oss
Why LogStash:
Depending on the environment, LogStash can act as an intermediary to process data from various sources and send it to a specified destination, called an output. I can think of three simple examples for the RSA use case. The first example is sending data from the data lake to Netwitness. The second is sending data from Netwitness to other sources. Finally, it can also be configured to collect data from various sources and send that data to Netwitness or the data lake.
https://en.wikipedia.org/wiki/Data_lake
RSA LogStash Components:
As mentioned above there are several scenarios. Depending on the configuration ether the Codec or Export Connector will be used.
LogStash Codec:
https://community.rsa.com/docs/DOC-113777
The Codec is used forward Logstash events to the NetWitness Platform in RFC-5424 format, you need to install the NetWitness codec on your system and refer to it in your output plugin configuration.
Example:
Data Lake/Warehouse, Syslog, etc --> LogStash --> RSA Decoder
LogStatsh Export Connector:
https://community.rsa.com/docs/DOC-114533
Logstash Output plugin to send the input events to a data warehouse destination.
Example:
RSA Decoder --> LogStash --> Data Lake/Warehouse
Supporting Documentation:
RSA NetWitness 11.5 Logstash Codec Download
Export Connector Install: Configure Logstash Output Plugin
Logstash Integration Guide for RSA NetWitness® Platform 11.5
The Process:
The process is fairly simple but LogStash is a product external to RSA and managed by Elastic. Because of this, it will be necessary to download LogStash and create its own virtual machine. This demonstration will provide a foundation to learn more about the process and how everything interacts.
Preparation for the demonstration:
Download the CentOS 7 iso.
Download LogStash if you want to manually install it. This demo will pull it directly using the rpm –import method.
Download Logstash Free • Get Started Now | Elastic
Once CentOS is installed, the following items are going to be required for the Demo. The actual install, may be different, so please note this.
Minimal Install of Centos 7
The minimal install is all that the demo requires
add jre for keytool
Keytool is required for the demo certificate process
yum install java-11-openjdk-devel
yum install mlocate
mlocate is used to quickly search for files
----> Begin initial setup
Demonstration
Download and install the public signing key:Insert into logstash.repo
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
vi /etc/yum.repos.d/logstash.repoCopy and paste into the new file
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
The repository is ready for use. You can install it with:
yum install logstash
systemctl enable logstash.service #Adds to startup
reboot
Check the LogStash status to make sure the LogStash service automatically starts after the reboot
systemctl status logstashDownload and copy the connector to tmp (WinSCP) - netwitness-export-connector-1.1.0.zip
systemctl stop logstash
cd /usr/share/logstash
bin/logstash-plugin install file:///tmp/netwitness-export-connector-1.1.0.zip
Validate the connector is installed
bin/logstash-plugin listAdd the following to the conf file (this initiates data pull via the decoder api on the decoder 50104)
vi /etc/logstash/conf.d/netwitness-192.168.2.121-input.confinput {
netwitness_export_connector {
host => "192.168.x.x" #Decoder IP Address
username => "admin" #API username
password => "your_password" #API password
decoder_type => "decoder"
}
}Clean up the input.conf to make sure there are no characters that cannot be seen
sed -i -e 's/\r$//' /etc/logstash/conf.d/netwitness-192.168.x.x-input.confRestart LogStash to load the input.conf
systemctl start logstash
tailf /var/log/logstash/logstash-plain.log... if you see regex errors check E.3 again
Look for a different sessionid
Test with TCPDump
cd /tmp
tcpdump -i eth0 -A -nvvv port 50004 -s 65535 -w logstashDump.txt
tcpdump -r logstashDump.txt
Example History for the export connector
[root@localhost conf.d]# history
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
vi /etc/yum.repos.d/
vi /etc/yum.repos.d/logstash.repo
yum install logstash
systemctl enable logstash.service
reboot
systemctl status logstash
systemctl stop logstash
cd /usr/share/logstash
bin/logstash-plugin install file:///tmp/netwitness-export-connector-1.1.0.zip
systemctl start logstash
----> Begin with the certificate section
Demonstration: No Audio
On the LogStash Server
If there is an existing certificate authority proceed to step 2. (Step 1) - The certificates can be called what ever fits the environment.
openssl genrsa -out CA-key.pem 2048
openssl req -new -key CA-key.pem -x509 -days 2000 -out CA-cert.pem
Resume Certificate Process (Step 2)
openssl genrsa -out server-key.pem 2048
openssl req -new -key server-key.pem -out signingReq.csr (set a password)
openssl x509 -req -days 1000 -in signingReq.csr -CA CA-cert.pem -CAkey CA-key.pem -CAcreateserial -out server-cert.pem
openssl pkcs12 -export -in server-cert.pem -inkey server-key.pem -certfile CA-cert.pem -out logstash-input-netwitness.p12 (set export password)
mkdir /etc/pki/logStashDecoder (on the logstash server)
On the decoder
scp /etc/pki/nw/trust/truststore.pem root@192.168.x.x:/etc/pki/logStashDecoderif known_host contains a key for this host you may want to delete it.
vi ~/.ssh/known_hostsLogStash Server
keytool -importcert -keystore logstash-input-netwitness.p12 -trustcacerts -alias nw-inter -file /etc/pki/logStashDecoder/truststore.pem -storetype PKCS12 (enter password)
Trust this certificate: yes
cp logstash-input-netwitness.p12 /etc/logstash
chown logstash:logstash /etc/logstash/logstash-input-netwitness.p12
Curl to the Decoder API
curl -X POST -d server-cert.pem https://192.168.2.121:50104/sys/trustpeer
curl -X POST -d server-cert.pem https://192.168.2.121:50104/sys/caupload
alternative to the curl command above - If using the rest interface is preferred (trustpeer and caupload)
https://192.168.x.x:50104/sys/trustpeer
https://192.168.x.x:50104/sys/caupload
copy and paste the server-cert.pem
Check the certificate exists on the Decoder
cd /etc/netwitness/ng/decoder/trustpeersChange the keystore password on logstash - current password is changeit
keytool -storepasswd -keystore /usr/lib/jvm/java-11-openjdk-11.0.9.11-0.el7_9.x86_64/lib/security/cacerts
Untrusted certificate from above (ex. your_password)
keytool -importcert -file /etc/pki/logStashDecoder/truststore.pem -keystore /usr/lib/jvm/java-11-openjdk-11.0.9.11-0.el7_9.x86_64/lib/security/cacerts -alias nw-core-cert -storepass your_password
systemctl restart logstash tailf /var/log/logstash/logstash-plain.log
Please let me know your thoughts or suggestions, as this is a work in progress.
