Troubleshoot Feeds

Overview

The purpose of the feed generator is to generate a mapping of an event source to the list of groups to which it belongs.

If you have an event source from which you are collecting messages, and it is not displayed in the correct event source groups, this topic provides background and information to help you track down the problem.

Details

The ESM Feed maps multiple keys to single value. It maps the DeviceAddress, Forwarder, and DeviceType attributes to groupName.

The purpose of the ESM feed is to enrich event source meta with the groupName collected on the Log Decoder.

How it Works

The feed generator is scheduled to update every minute. However, it is triggered only if there are any changes (create, update, or delete) in event sources or groups.

It generates a single feed file with event-source-to-group mapping, and pushes the same feed to all of the Log Decoders that are connected to NetWitness.

Once the feed file is uploaded on the Log Decoders, for any new events, it enriches events meta data with groupName, and appends this groupName to logstats.

Once the groupName is in logstats, the ESM Aggregator groups information and sends it to ESM. At this point, you should see the Group Name column under the Event Source Monitoring tab.

The entire process can take some time. Therefore, you may need to wait for several seconds after you add a new group or event source, before the Group name is displayed.

Note: If the event source type attribute changes when the feed is updated, NetWitness adds a new logstats entry rather than updating the existing one. Thus, there will be two different logstats entries in logdecoder. Previously existing messages would have been listed under the previous type, and all new messages are logged for the new event source type.

Feed File

The format of the feed file is as follows:

DeviceAddress, Forwarder, DeviceType, GroupName

The DeviceAddress is either ipv4, ipv6, or hostname, depending on which of these have been defined for the event source.

The following is a sample of the feed file:

"12.12.12.12","d6","NETFLOW","grp1"
"12.12.12.12","ld4","netflow","grp1"
"12.12.12.12","d6","netfow","grp1"
"0:E:507:E6:D4DB:E:59C:A","10.25.50.243","apache","Apachegrp"
"1.2.3.4","LCC","apache","Apachegrp"
"10.100.33.234","LC1","apache","Apachegrp"
"10.25.50.248","10.25.50.242","apache","Apachegrp"
"10.25.50.251","10.25.50.241","apache","Apachegrp"
"10.25.50.252","10.25.50.255","apache","Apachegrp"
"10.25.50.253","10.25.50.251","apache","Apachegrp"
"10.25.50.254","10.25.50.230","apache","Apachegrp"
"10.25.50.255","10.25.50.254","apache","Apachegrp"
"13.13.13.13","LC1","apache","Apachegrp"
"AB:F255:9:8:6C88:EEC:44CE:7",,"apache","Apachegrp"
"Appliance1234",,"apache","Apachegrp"
"CB:F255:9:8:6C88:EEC:44CE:7","10.25.50.253","apache","Apachegrp"

Troubleshooting

You can check the following items to narrow down where the problem is occurring.

Feed File Existence

Verify that the feeds ZIP archive exists in the following location:

/opt/rsa/sms/esmfeed.zip

Do not modify this file.

Group Meta Populated on LD

Verify that the group meta is populated on the Log Decoder. Navigate to the Log Decoder REST and check logstats:

http://LogDecoderIP:50102/decoder?msg=logStats&force-content-type=text/plain

This is a sample logstats file with group information:

device=apache  forwarder=NWAPPLIANCE10304  source=1.2.3.4  count=338  lastSeenTime=2015-Feb-04 22:30:19  lastUpdatedTime=2015-Feb-04 22:30:19  groups=IP1234Group,apacheGroup
device=apachetomcat  forwarder=NWAPPLIANCE10304  source=5.6.7.8  count=1301  lastSeenTime=2015-Feb-04 22:30:19  lastUpdatedTime=2015-Feb-04 22:30:19  groups=AllOtherGroup,ApacheTomcatGroup

In the above text, the group information is bold.

Device Group Meta on Concentrator

Verify that the Device Group meta data exists on the Concentrator, and that events have values for the device.group field.

netwitness_esm_tbl_devgrp1.png

netwitness_esm_tbl_devgrp2.png

SMS Log File

Check the SMS log file in the following location to view informational and error messages: /opt/rsa/sms/logs/sms.log

The following are examples informational messages:

Feed generator triggered...
Created CSV feed file.
Created zip feed file.
Pushed ESM Feed to LogDeocder : <logdecoder IP>

The following are examples of error messages:

Error creating CSV File : <reason>Unable to push the ESM Feed: Unable to create feed zip archive.
Failed to add Group in CSV: GroupName: <groupName> : Error: <error>  
Unable to push the ESM Feed: CSV file is empty, make sure you have al-least on group with al-least one eventsource.   
Unable to push the ESM Feed: No LogDecoders found.
Unable to push the ESM Feed: Unable to push feed file on LogDecoder-<logdecoderIP>Unable to push the ESM Feed: admin@<logdecoderIP>:50002/decoder/parsers received error: The zip archive "/etc/netwitness/ng/upload/<esmfeedfileName>.zip" could not be opened  
Unable to push the ESM Feed: <reason>

Verify Logstats Data is Getting Read and Published by ESMReader and ESMAggregator

These are the steps to verify that logstats are collected by collectd and published to Event Source Management.

ESMReader

  1. On LogDecoders add the debug "true" flag in /etc/collectd.d/NwLogDecoder_ESM.conf: 
#
# Copyright (c) 2014 RSA The Security Division of EMC
#
<Plugin generic_cpp> PluginModulePath "/usr/lib64/collectd"
    debug "true"
    <Module "NgEsmReader" "all"> port "56002"
        ssl       "yes"
        keypath   "/var/lib/puppet/ssl/private_keys/d4c6dcd4-6737-4838-a2f7-ba7e9a165aae.pem"
        certpath  "/var/lib/puppet/ssl/certs/d4c6dcd4-6737-4838-a2f7-ba7e9a165aae.pem"
        interval  "600"
        query     "all"
        <stats></stats></Module><Module "NgEsmReader" "update"> port "56002" ssl "yes" 
        keypath   "/var/lib/puppet/ssl/private_keys/d4c6dcd4-6737-4838-a2f7-ba7e9a165aae.pem"
        certpath  "/var/lib/puppet/ssl/certs/d4c6dcd4-6737-4838-a2f7-ba7e9a165aae.pem"
        interval  "60"
        query     "update"
        <stats></stats></Module></Plugin>
  1. Run the command:
    collectd service restart
  2. Run the following command:
    tail –f /var/log/messages | grep collectd
    Verify that ESMReader is reading logstats and there are no errors. If there are any read issues, you will see errors similar to the following:     
​Apr 29 18:47:45 NWAPPLIANCE15788 collectd[14569]: DEBUG: NgEsmReader_all: error getting ESM data for field "groups" from logstat device=checkpointfw1 forwarder=PSRTEST source=1.11.51.212. Reason: <reason>Apr 29 18:58:36 NWAPPLIANCE15788 collectd[14569]: DEBUG: NgEsmReader_update: error getting ESM data for field "forwarder" from logstat device=apachetomcat source=10.31.204.240. Reason: <reason>​

ESMAggregator

  1. On NetWitness, uncomment the verbose flag in /etc/collectd.d/ESMAggregator.conf:
    # ESMAggregator module collectd.conf configuration file
    #
    # Copyright (c) 2014 RSA The Security Divsion of EMC
    #

    <Plugin generic_cpp>    PluginModulePath "/usr/lib64/collectd"

    <Module "ESMAggregator">
            verbose 1
            interval "60"
            cache_save_interval "600"
            persistence_dir "/var/lib/netwitness/collectd"
    </Module>    </Plugin>
  1. Run the following command:
    collectd service restart.
  2. Run the following command:
    run “tail –f /var/log/messages | grep ESMA
    ​Look for for ESMAggregator data and make sure your logstat entry is available in logs.

Sample output:

Mar  1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[0] logdecoder[0] = d4c6dcd4-6737-4838-a2f7-ba7e9a165aae
Mar  1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[1] logdecoder_utcLastUpdate[0] = 1425174451
Mar  1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[2] groups = Cacheflowelff,Mixed
Mar  1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[3] logdecoders = d4c6dcd4-6737-4838-a2f7-ba7e9a165aae
Mar  1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[4] utcLastUpdate = 1425174451
Mar  1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: Dispatching ESM stat NWAPPLIANCE15788/esma_update-cacheflowelff/esm_counter-3.3.3.3 with a value of 1752 for NWAPPLIANCE15788/cacheflowelff/esm_counter-3.3.3.3 aggregated from 1 log decoders
Mar  1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[0] logdecoder[0] = 767354a8-5e84-4317-bc6a-52e4f4d8bfff
Mar  1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[1] logdecoder_utcLastUpdate[0] = 1425174470
Mar  1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[2] groups = Cacheflowelff,Mixed
Mar  1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[3] logdecoders = 767354a8-5e84-4317-bc6a-52e4f4d8bfff
Mar  1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: MetaData[4] utcLastUpdate = 1425174470
Mar  1 02:32:08 NWAPPLIANCE15936 collectd[11203]: ESMAggregator: Dispatching RRD stat NWAPPLIANCE15788/esma_rrd-cacheflowelff/esm_counter-3.3.3.3 with a value of 1752 for NWAPPLIANCE15788/cacheflowelff/esm_counter-3.3.3.3 aggregated from 1 log

Configure JMX Feed Generator Job Interval

Although the feed generation job is scheduled to execute every minute by default, you can change this by using jconsole.

To change the feed generator job interval:

  1. Open jconsole for the SMS service.
  2. On the MBeans tab, navigate to com.rsa.netwitness.sms > API > esmConfiguration > Attributes.
  3. Modify the value for the property FeedGeneratorJobIntervalInMinutes.
  4. Go to Operations under the same navigation tree, an click commit(). This persists the new value in the corresponding json file under /opt/rsa/sms/conf, and uses the value if SMS is restarted.

​Setting a new value reschedules the feed generator job for the new interval.