NwLogDecoder Syslog Warning "Unidentified content from"

[Anonymous]

We try to send demo data from ArcSight to Security Analytics LogDecoder.

tcpdump shows that the logdecoder device receives the messages from network, but  NwLogDecoder don't decode it.

NwLogDecoder produces following warning in /var/log/messages:

Jan 16 11:49:00 LogDecoder nw[6462]: [SYSLOG] [warning] Unidentified content from xxx.xxx.xxx.xxx received on receiver: 'CEF:0|Test|Test|Test|Test|Test|Test|Test'

Could anyone help?

[Anonymous]

I have actually never seen that message on my decoder.

But it might be because the CEF format is wrong?  It is looking for a time, hostname than CEF:0

content="<event_time_string><hostname>CEF:<cefversion>|<devvendor>|<product>|<version>|<event_type>|<event_description>|<severity>|<!payload>"/>

[Anonymous]

Wireshark Trace for network interface shows the raw syslog cef content:

14:58:04.370311 IP 192.168.xxx.xxx.35867 > rsa_logdecoder.syslog: [|syslog]

E..... .@..........^......d.CEF:0|Check Point|FireWall-1|4.1|a ....cutted

So we should get following header

Jan 15 14:58:04 192.168.xxx.xxx.35867 CEF: 0|Check Point|FireWall-1|4.1|a ....cutted

[Anonymous]

Interesting.

Just to verify on your log decoder the CEF parser is turned on?

And you have a cef.xml located at /etc/netwitness/ng/envision/etc/devices/cef?

[Anonymous]

Yes cef.xml is installed.

If I start rsyslog und start nwlogdecoder the nwlogdecoder writes the "[SYSLOG] [warning] Unidentified content"

into /var/log/messages.

If I stop rsyslog, the nwlogdecoder doesnt capture any syslog.

If I start rsyslog and stop nwlogdecoder, the CEF entries are written correkt into /var/log/messages.

[RSAAdmin · Accepted solution]

The demo data sent from ArcSight is not in a format supported by Log Decoder.

Here is a quick classification of the various types of payloads that can arrive on Log Decoder port 514 and how Log Decoder treats them.

1. Syslog

   Example:

       <1> %PIX-1-105032: LAN Failover interface is down   

   A valid syslog supported by Log Decoder must begin with the PRI (i.e. the priority) as defined in Section 4.1.1 of RFC 3164. In the example, above, the string <1> is the PRI part.

2. Log forwarded by Log Collector to Log Decoder

   Example:

       [][FOO-VLC1][10.0.0.1][1412278173101][] %PIX-1-105032: LAN Failover interface is down   

   When logs are collected by Log Collector (nwlogcollector) and forwarded to Log Decoder (nwlogdecoder) on port 514, the logs are in this format. The tokens within square-brackets in the beginning contain sender information which is used by the Log Decoder to generate related metas. For example, for the Log Collector tags in this example, we'll see two metas generated: lc.cid = FOO-VLC1, device.ip = 10.0.0.1.

3. Unknown log

   Example:

       <1> hello world   

   This is a valid syslog from Log Decoder's perspective, so it accepts this log. But this log does not contain valid content that is supported by the Content 2.0 device parsers at /etc/netwitness/ng/envistion/etc/devices. So although, Log Decoder accepts these logs, we'll not see many metas generated.  In particular, we'll see device.type = unknown meta indicating that this log is from a device type not supported by the Log Decoder parsers.

4. Unsupported log (Unidentified content)

   Example:

       %PIX-1-105032: LAN Failover interface is down   

   This is not a valid log supported by Log Decoder because the PRI part (as explained in the first point) is missing in this log. When Log Decoder receives this log, it rejects this log with the following warning: [SYSLOG]  Unidentified content from 127.0.0.1 received on receiver: '%PIX-1-105032: LAN Failover interface is down'.

The demo data sent from ArcSight belongs to the category of unsupported log. It cannot be directly fed to Log Decoder in its current form because the PRI part is missing from the log. Such logs may have to be fed to Log Decoder via Log Collector (perhaps by using file reader) but I am not an expert in Log Collector to be able to tell you for sure if and how it may work.

If you want to quickly test the various types of logs, you can try injecting them directly to Log Decoder by running the following commands on the Log Decoder and see what metas they generate by running the following commands.

echo " %PIX-1-105032: LAN Failover interface is down 1" > /dev/tcp/127.0.0.1/514

echo "[][FOO-VLC1][10.0.0.1][1412278173101][] %PIX-1-105032: LAN Failover interface is down 2" > /dev/tcp/127.0.0.1/514

echo " hello world" > /dev/tcp/127.0.0.1/514

echo "%PIX-1-105032: LAN Failover interface is down" > /dev/tcp/127.0.0.1/514

Thanks,

Susam

[Anonymous]

Very many thanks for the detailed description!

Maybe we will get the LogCollector scenario configured. For this we need a remote collector with syslog collecting functionality.

[Anonymous]

I have tried a CEF Message like:

echo "CEF: 0|Unix|Unix|1|Demo|Event Description|Low|" > /dev/tcp/127.0.0.1/514

But the CEF Parser from Live doesn't catch this header. Can RSA provide a better CEF parser for logs?

[RSAAdmin]

Did you feed the CEF log to Log Decoder, a VLC or another syslog service?

If you feed a CEF log to Log Decoder as specified in your command, i.e.

echo "CEF: 0|Unix|Unix|1|Demo|Event Description|Low|" > /dev/tcp/127.0.0.1/514

then this log would be rejected by Log Decoder with the following warning.

[SYSLOG]  Unidentified content from 127.0.0.1 received on receiver: 'CEF: 0|Unix|Unix|1|Demo|Event Description|Low|'

But if you have setup VLC and fed this log to VLC which forwarded this log to Log Decoder and it appeared as device.type=unknown or it appeared as device.type=cef but msg.id meta was missing, in Investigation, then you are right that the CEF parser failed to parse it.

In such a case, you might want to report the issue to RSA Customer Support with the log that reproduces the issue, so that the parser can be improved.

[Anonymous]

I feed following line to LogDecoder, and it works:

echo "<1> Jan 23 16:00:00 CEF: 0|Unix|Unix|1|Demo|Event Description|Low|" > /dev/udp/127.0.0.1/514

Alternatively I use the NwLogPlayer tool. For this a can set the Priority Flag in my testdata.