2016-07-12 05:33 AM
It seems that either RSA Security Analytics is not using the common fw1-loggrabber to fetch Check Point FW-1 events, or the tool is at least heavily modified. What I am looking for is the list of all fields extracted by Check Point event collection in the cases of both Security and Audit events. As you probably know, the events collected by the Check Point collection module are comma-separated values, with multiple fields usually containing an empty value. This leads me to believe that the fetcher is actually always fetching all the fields, and because not all event types contain all fields, the resulting raw events contain empty values. What I would like to know is the list of all the fields that are fetched, and whether the ordering of the fields in the resulting raw event is always the same.
2016-07-12 07:57 AM
The process used by the Logcollector service to collect checkpoint events can be run manually from the Logcollector command line to check how logs will look like. This service options can be passed in a configuration file or directly while running the service as follows:
NwCheckpointProcess --ip <CheckpointServerIP> --name <OPSEC_ObjectName> --port 18184 --sdn <SDN> --cdn <CDN> --cen <CEN> --kfp /etc/netwitness/ng/truststore/<CertificateName>.p12 --count <NumberOfLogs> --time 120 --timeout 30 --audit
This is then passed to the Logdecoder XML parser in /etc/netwitness/ng/envision/etc/devices/checkpointfw1/v20_checkpointfw1msg.xml
2016-07-13 05:50 AM
Thank you for the example; I was able to get that command working without any hiccups. However, correct me if I am wrong, but the output (at least considering the format NwCheckpointProcess) is being post-processed somewhere internally before passing it into the XML parser, as the NwCheckpointProcess at least print the events in key-value pair format, whereas the XML parser expects basic comma-separated format without the keys. As a result, a NwCheckpointProcess output with, say, 20 key-value pairs might end up into an XML message definition with 50 fields, where 30 of those fields are simply passed into temporary/transient variables, which do not get mapped into nwdb, as the values do not exists in the fetched event.
So to me it seems that if we have a NwCheckpointProcess output of
key_1=value_1,key_2=value_2,key_3=value_3,...,key_n=value_n
it will get internally mapped to the following format
value_1,value_2,value_3,...,value_m
where n is most probably smaller than m, thus leaving empty values in the latter format.
Then again I might be wrong and the underlying format is the latter all along, but NwCheckpointProcess simply prints things out in the former format. I am still interested in any listing (e.g. a file where the mapping between the two formats is defined, if that even is reality) that would correspond to the latter format, i.e. what is the list of all keys and what is their ordering in the latter format. From the XML parser alone this is really difficult if not impossible to see, since no message definition is using m keys, and all non-used keys/values are handled with variables such as "fld80", thus masking out what the field would contain were it not empty.
2016-07-13 10:38 AM
Hi Tomi,
You are right. The logcollector maps those variables to form the Syslog message that is then passed to the logdecoder. This mapping is not available in flat files and it is not possible to change it.
2016-07-14 02:08 AM
I am fine with not changing the mapping. However, is it possible to obtain documentation on the sequence
value_1, value_2, value_3, ..., value_m
so that all the fields would be named and the ordering would correlate to the XML parser? By motivation behind this is to see whether the whole parsing process for Check Point FW-1 could be done with significantly fewer message definitions in the parser XML.
2016-07-14 12:29 PM
Hi Tomi,
The Logcollector would transform and send the name-value pairs it gets from the nwcheckpointprocess to the the logdecoder using protobuf (http://code.google.com/p/protobuf/) to form the format the Content 2.0 parser is expecting. This transform is defined in /etc/netwitness/ng/logcollection/content/transform/checkpoint/chkpntevent.xml on the logcollector.
I hope that helps
Best regards,
Abdelrahman
2016-07-15 03:13 AM
This is exactly what I was looking for. Much appreciated!