Troubleshooting Feeds
The purpose of the feed generator is to generate a mapping of an event source to the list of groups to which it belongs.
If you have an event source from which you are collecting messages, and yet it is not displayed in the correct event source groups, then this topic provides background and information to help you track down the problem.
Details
The ESM Feed maps multiple keys to single value. It maps the DeviceAddress, Forwarder, and DeviceType attributes to groupName.
The purpose of the ESM feed is to enrich event source Meta with the groupName collected on the Log Decoder.
How it Works
The feed generator is scheduled to update every minute. However, it is triggered only if there are any changes (create, update, or delete) in event sources or groups.
It generates a single feed file with event source to group mapping, and pushes the same feed to all of the Log Decoders that are connected to NetWitness.
After the feed file is uploaded on the Log Decoders, for any new events, it enriches events Meta data with groupName, and appends this groupName to logstats.
After the groupName is in logstats, the ESM Aggregator groups information and sends it to ESM. At this point, you should see the Group Name column under the Event Source Monitoring tab.
The entire process can take some time. Therefore, you may need to wait for several seconds after you add a new group or event source, before the Group name is displayed.
Note: If the event source type attribute changes when the feed is updated, NetWitness adds a new logstats entry, rather than updating the existing one. Thus, there will be two different logstats entries in logdecoder. Previously existing messages would have been listed under the previous type, and all new messages are logged for the new event source type.
Feed File
The format of the feed file is as follows:
DeviceAddress, Forwarder, DeviceType, GroupName
The DeviceAddress is either ipv4, ipv6, or hostname, depending on which of these have been defined for the event source.
The following is a sample of the feed file:
"12.12.12.12","d6","NETFLOW","grp1"
"12.12.12.12","ld4","netflow","grp1"
"12.12.12.12","d6","netfow","grp1"
"0:E:507:E6:D4DB:E:59C:A","10.25.50.243","apache","Apachegrp"
"1.2.3.4","LCC","apache","Apachegrp"
"10.100.33.234","LC1","apache","Apachegrp"
"10.25.50.248","10.25.50.242","apache","Apachegrp"
"10.25.50.251","10.25.50.241","apache","Apachegrp"
"10.25.50.252","10.25.50.255","apache","Apachegrp"
"10.25.50.253","10.25.50.251","apache","Apachegrp"
"10.25.50.254","10.25.50.230","apache","Apachegrp"
"10.25.50.255","10.25.50.254","apache","Apachegrp"
"13.13.13.13","LC1","apache","Apachegrp"
"AB:F255:9:8:6C88:EEC:44CE:7",,"apache","Apachegrp"
"Appliance1234",,"apache","Apachegrp"
"CB:F255:9:8:6C88:EEC:44CE:7","10.25.50.253","apache","Apachegrp"
Troubleshooting Feeds
You can check the following items to narrow down where the problem is occurring.
Feed File Existence
Verify that the feeds ZIP archive exists in the following location:
/opt/rsa/sms/esmfeed.zip
Do not modify this file.
Group Meta Populated on LDGroup Meta Populated on LD
Verify that the group meta is populated on the Log Decoder. Navigate to the Log Decoder REST and check logstats:
http://LogDecoderIP:50102/decoder?msg=logStats&force-content-type=text/plain
This is a sample logstats file with group information:
device=apache forwarder=NWAPPLIANCE10304 source=1.2.3.4 count=338 lastSeenTime=2015-Feb-04 22:30:19 lastUpdatedTime=2015-Feb-04 22:30:19
groups=IP1234Group,apacheGroup
device=apachetomcat forwarder=NWAPPLIANCE10304 source=5.6.7.8 count=1301 lastSeenTime=2015-Feb-04 22:30:19 lastUpdatedTime=2015-Feb-04 22:30:19
groups=AllOtherGroup,ApacheTomcatGroup
In the above text, the group information is bolded.
Device Group Meta on Concentrator
Verify that the Device Group meta exists on the Concentrator, and that events have values for the device.group field.
SMS Log File
Check the SMS log file in the following location to view informational and error messages: /opt/rsa/sms/logs/sms.log
The following are example informational messages:
Feed generator triggered... Created CSV feed file. Created zip feed file. Pushed ESM Feed to LogDeocder : <logdecoder IP>
The following are example error messages:
Error creating CSV File : <reason>Unable to push the ESM Feed: Unable to create feed zip archive.
Failed to add Group in CSV: GroupName: <groupName> : Error: <error>
Unable to push the ESM Feed: CSV file is empty, make sure you have al-least on group with at least one eventsource.
Unable to push the ESM Feed: No LogDecoders found.
Unable to push the ESM Feed: Unable to push feed file on LogDecoder-<logdecoderIP>Unable to push the ESM Feed: admin@<logdecoderIP>:50002/decoder/parsers received error: The zip archive "/etc/netwitness/ng/upload/<esmfeedfileName>.zip" could not be opened
Unable to push the ESM Feed: <reason>
Verify Logstats data is getting Read & Published by ESMReader & ESMAggregator
These are the steps to verify that logstats are collected by collectd and published to Event Source Management.
ESMReader
-
On Log Decoders add debug "true" flag in in /etc/collectd.d/logdecoder-esm.conf:
#
# Copyright (c) 2014 RSA The Security Division of EMC
#
<Plugin generic_cpp> PluginModulePath "/usr/lib64/collectd"
debug "true"
<Module "NgEsmReader" "all">
port "56002"
ssl "yes"
keypath "/etc/pki/nw/node/node-key.pem"
certpath "/etc/pki/nw/node/node-cert.pem"
interval "600"
query "all"
<stats>
</stats>
</Module>
<Module "NgEsmReader" "update">
port "56002"
ssl "yes"
keypath "/etc/pki/nw/node/node-key.pem"
certpath "/etc/pki/nw/node/node-cert.pem"
interval "60"
query "update"
<stats>
</stats>
</Module>
</Plugin> -
Run the command:
collectd service restart
-
Run the following command:
tail –f /var/log/messages | grep collectd
Verify that ESMReader is reading logstats and there are no errors. If there are any read issues, you will see errors similar to the following:
Apr 29 18:47:45 NWAPPLIANCE15788 collectd[14569]: DEBUG: NgEsmReader_all: error getting ESM data for field "groups" from logstat device=checkpointfw1 forwarder=PSRTEST source=1.11.51.212. Reason: <reason>Apr 29 18:58:36 NWAPPLIANCE15788 collectd[14569]: DEBUG: NgEsmReader_update: error getting ESM data for field "forwarder" from logstat device=apachetomcat source=10.31.204.240. Reason: <reason>
ESMAggregator
-
On NetWitness, uncomment the verbose flag in /etc/collectd.d/ESMAggregator.conf:
# ESMAggregator module collectd.conf configuration file
#
# Copyright (c) 2014 RSA The Security Divsion of EMC
#
<Plugin generic_cpp>
PluginModulePath "/usr/lib64/collectd"
<Module "ESMAggregator">
verbose 1
interval "60"
cache_save_interval "600"
persistence_dir "/var/lib/netwitness/collectd"
</Module>
</Plugin> -
Run the following:
collectd service restart.
-
Run the following command:
run “tail –f /var/log/messages | grep ESMA
Look for ESMAggregator data and make sure your logstat entry is available in logs.
Sample output:
Mar 1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[0] logdecoder[0] = d4c6dcd4-6737-4838-a2f7-ba7e9a165aae
Mar 1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[1] logdecoder_utcLastUpdate[0] = 1425174451
Mar 1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[2] groups = Cacheflowelff,Mixed
Mar 1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[3] logdecoders = d4c6dcd4-6737-4838-a2f7-ba7e9a165aae
Mar 1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[4] utcLastUpdate = 1425174451
Mar 1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: Dispatching ESM stat NWAPPLIANCE15788/esma_update-cacheflowelff/esm_counter-3.3.3.3 with a value of 1752 for NWAPPLIANCE15788/cacheflowelff/esm_counter-3.3.3.3 aggregated from 1 log decoders
Mar 1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[0] logdecoder[0] = 767354a8-5e84-4317-bc6a-52e4f4d8bfff
Mar 1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[1] logdecoder_utcLastUpdate[0] = 1425174470
Mar 1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[2] groups = Cacheflowelff,Mixed
Mar 1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[3] logdecoders = 767354a8-5e84-4317-bc6a-52e4f4d8bfff
Mar 1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[4] utcLastUpdate = 1425174470
Mar 1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: Dispatching RRD stat NWAPPLIANCE15788/esma_rrd-cacheflowelff/esm_counter-3.3.3.3 with a value of 1752 for NWAPPLIANCE15788/cacheflowelff/esm_counter-3.3.3.3 aggregated from 1 log
Configure JMX Feed Generator Job Interval
Although the feed generation job is scheduled to execute every minute by default, you can change this by using jconsole, if necessary.
To change the feed generator job interval:
- Open jconsole for the SMS service.
- On the MBeans tab, navigate to com.rsa.netwitness.sms > API > esmConfiguration > Attributes.
- Modify the value for the property FeedGeneratorJobIntervalInMinutes.
- Go to Operations under the same navigation tree, an click commit(). This persists the new value in the corresponding json file under /opt/rsa/sms/conf, and uses the value if SMS is restarted.
Setting a new value reschedules the feed generator job for the new interval.
