2015-02-03 12:54 PM
Does anyone documentation, or have the ability on how the xml Log parsing works in SA? I would like to analyze how the XML fully works in detail, along with easily modifying / creating new parsers for our internal use. I am having a hard time currently trying to read these xml parsers, let along creating them.
2015-02-04 12:35 PM
sorry, you are right, well the bad news is that RSA doesn't have too much information.
You have two roads, one is to create log parsers for devices not supported by RSA, you have to use the ESI Tool to create the xml and config files. And then upload the files to the log decoder. There is a post here called "Security Analytics Log Parser" you can take a look on how to upload those file to SA.
The other path is to modify the RSA parser (this is not supported)
There are some key files
table-map.xml
v20_eventsourcename.xml file
If it remember correctly they are inside this directory
/etc/netwitness/ng/etc/
The parser is inside a directory with the name
In the table map file you have the conversion variables from envision to SA. (yep it's correct) the parser is the same as envision. When a variable has transient flag it is not stored on the system, and when it's None it is stored on the system /this modification will affect all parsers that use that particular variable). It is recommended to use a table-map-custom.xml for all changes in that file.
When a log to SA is received it uses the log parser which has the envision variables and process the log according to the message id received, then translate the variables using table map so SA saves them on their respective meta.
if you want to modify any parser for any log device for example a variable is not parsing and you want it, you have to modify the variable used in the parser (usually RSA call the variables <field1>, <field2>...<fieldX>) so you have to change that to something the system saves (for example ec_theme, replace field1 for ec_theme) remember you have to modify using the envision variables not the SA variable (with the dot).
And the changes has to be done in each message id, I usually do a global replace with VI
It's a very painful situtation but this is what it is.
Also what I do , it copy the message from the parser and the received from the event source into notepad, remove the ;> and < to the actual symbols and the replace them to commas, then save it as CSV and the open it in excel so it has every field separated by columns, and the compare that to see what field I have to change,
After that changes you have to restart nwlogdecoder service, if everything is ok, the log should tell you that the parser for the certain source is parsed is loaded correctly.
let me know if this helps....
2015-02-03 04:05 PM
2015-02-04 10:49 AM
Thanks, that document appears to be focused towards packet parsers, not log parsers.
2015-02-04 12:35 PM
sorry, you are right, well the bad news is that RSA doesn't have too much information.
You have two roads, one is to create log parsers for devices not supported by RSA, you have to use the ESI Tool to create the xml and config files. And then upload the files to the log decoder. There is a post here called "Security Analytics Log Parser" you can take a look on how to upload those file to SA.
The other path is to modify the RSA parser (this is not supported)
There are some key files
table-map.xml
v20_eventsourcename.xml file
If it remember correctly they are inside this directory
/etc/netwitness/ng/etc/
The parser is inside a directory with the name
In the table map file you have the conversion variables from envision to SA. (yep it's correct) the parser is the same as envision. When a variable has transient flag it is not stored on the system, and when it's None it is stored on the system /this modification will affect all parsers that use that particular variable). It is recommended to use a table-map-custom.xml for all changes in that file.
When a log to SA is received it uses the log parser which has the envision variables and process the log according to the message id received, then translate the variables using table map so SA saves them on their respective meta.
if you want to modify any parser for any log device for example a variable is not parsing and you want it, you have to modify the variable used in the parser (usually RSA call the variables <field1>, <field2>...<fieldX>) so you have to change that to something the system saves (for example ec_theme, replace field1 for ec_theme) remember you have to modify using the envision variables not the SA variable (with the dot).
And the changes has to be done in each message id, I usually do a global replace with VI
It's a very painful situtation but this is what it is.
Also what I do , it copy the message from the parser and the received from the event source into notepad, remove the ;> and < to the actual symbols and the replace them to commas, then save it as CSV and the open it in excel so it has every field separated by columns, and the compare that to see what field I have to change,
After that changes you have to restart nwlogdecoder service, if everything is ok, the log should tell you that the parser for the certain source is parsed is loaded correctly.
let me know if this helps....
2015-02-04 02:04 PM
thanks for the response and information! Would you happen to know where I can get this ESI Tool, and any documentation regarding parser creation / tool usage?
2015-02-04 03:23 PM
Security Analytics Parser v2.0.zip
is one place to start. I'd also check out the packet parser book referenced above
2015-02-04 03:42 PM
you can find it on the SCOL site.
ESI stands for Event Source Integrator
2015-02-04 10:36 PM
if your just modifying the parser file. an ordinary decoder.parser /reload will suffice from the console.
2015-02-05 04:22 AM
On SCOL, the latest version of ESI is "2012, May 09 | RSA enVision EventSource Integrator 1.2.1"....
2015-02-06 02:05 AM
Does anybody knows if SA can accommodate and digest multi-lined logs? Has anyone pulled of a parser that can accept multi lined logs?