Event logs from Linux servers are mapping to multiple parsers

[VishamRawat]

Linux server event logs are mapping to crossbeamc parser, besides rhlinux as well.

Question - does a distinct event log [1 distinct event message] map to multiple parsers, or is it mapped to the 1st parser in the list of activated parsers? how does this work?

For eg., if the header of an event log is common to 2 or more parsers, will that particular event log map to both of them, or only 1 of them - based on some criterion?

Also, if I were to force-map the event source to rhlinux, how would I be guaranteed that the event logs being mapped to crossbeamc will start falling under rhlinux? what if they then start reporting as unknown?

If anybody could provide some clarity on how parsing works on a per-message or per-eventlog basis, and also what exactly happens when multiple parsers are matched. Do they match for the same eventlog message, or does a particular parser take precedence? and if such is the case [precedence], then how are we to know that a particular parser mapping is the result of choice between multiple parsers? what if we turn off the one wrongly matching [eg. crossbeamc], and those event log types [matching crossbeamc] then don't match where expected [eg. rhlinux]?

A little clarity on this would be very helpful.

Thanks!

RSA Security Analytics 10.6.4" data-type="space‌ RSA NetWitness Platform" data-type="space‌

[EricPartington]

disable all unnecessary log parsers to reduce potential problems.

As far as I understand log parsers are matched based on message header scores.  These are hidden in the 10.x code branch but in 11.x this is visible in the event sources tab.  The message with the best header match wins in terms of parsing that event.  Setting the parser mapping function forces that IP address to the parser(s). The order of the parser in the device mapping is important (set the same order for similar devices if there are multiple parsers for a device).  Devices that are mapped under the device parser function will not parse as unknown.  They will be forced to that device type (or best match if the wrong mapping is forced) and write device.type value and no meta extraction (word only meta or msg.id !exists)  

if you are on 10.6.4, consider getting to 11.x as there are improvements in the log parsing functions that will help reduce these issues.

[VishamRawat]

Thanks Eric.

Is there a document that gives an idea on the logic for message header scores?

The order of the parser in the device-mapping is important - this you say is for force-mapping, right? where multiple parsers are force-mapped to 1 event source, manually? although, I'm at a loss why this scenario would arise (multiple parsers force-mapped to a single event source)?

[EricPartington]

No public doc that describes the header logic

combinations commonly seen for multiple parsers to one device (it's forcing to the same device IP)

• Windows system with IIS
• OS level logging on linux system as well as application logging (firewall, proxy, ids)
• Windows + MSSQL
• Windows + MS DNS
• Linux + MySQL
• Linux + Apache

Pre-11.x you can use a number of searches and reports to determine purely based on the distinct count of parsers per device.ip where you might have issues that need to be manually resolved

if distinct(device.type) for device.ip is > 2

then you might need to look into the mapping

if distinct(device.type) for device.type >1 and device.type='rhlinux'

you should check into these to make sure you have the right parsers enabled (only) and any force mappings set as required as rhlinux can mis-parse due to the wide format of generic linux messages from a number of security devices that do a poor job of indicating the application.

The other solution is to break up you devices into groups of decoders based on type or collection method.  THen you can reduce your problem space even further by only enabling the troublesome parsers on an even more limited set of decoders.  Depends on the architecture and capabilities if this is an option, but I have seen it be very successful at customer sites.

[DavidEagen]

Ok, so we need to set up a decoder for every device type that needs to be parsed. How do we set up additional decoders on our existing system?

[EricPartington]

No you don't need a decoder for each device type, it certain cases it can help with architecture, load and other functions (including parsers) but is not a requirement.

Device mapping will help you resolve the issues listed above and which i think you have asked on another thread.  There are ways to help bulk create the mapping file if that's required also posted to another similar thread.