Article Number
000031068
Applies To
RSA Product Set: Security Analytics
RSA Product/Service Type: Security Analytics UI, Security Analytics Server, Decoder, Log Decoder
RSA Version/Condition: 10.5.0.0
Platform: CentOS
O/S Version: EL6
Issue
When using large csv files for custom feeds, the time taken for them to compile can be too great and result in a "Failed" message under the Feeds screen. This article explains a workaround.
The issue has been seen with csv feed files over 20MB. These may take over 20 minutes to compile on some systems.
Image descriptionThe steps below explain how to reproduce the issue.
- Create your custom feed as normal.
- Notice that for large CSV files the feed will fail to apply to the decoder.
Resolution
Run the following script on the SA Server. This can be run as a cronjob if desired.
The script will look through all your scheduled feeds, recompile them and then apply them. In the example script below
- 192.168.123.2 is the IP address of a Packet Decoder
- 192.168.123.3 is the IP address of a Log Decoder
Create a file with the following content and make it executable.
Be aware that the service account passwords are exposed in this file.
Make sure that password-less ssh connections have been set up between the SA Server and the log and packet decoders, so that the feeds can be copied over. For more information google the ssh-copy-id command.
find /var/lib/netwitness/uax/scheduler/ |grep xml >/tmp/feeds
for feed in $(cat /tmp/feeds)
do
FEEDDIR=$(dirname $feed)
FEEDNAME=$(basename $feed)
echo $FEEDDIR
echo $FEEDNAME
cd $FEEDDIR
NwConsole -c "feed create $FEEDNAME" -c "exit"
scp *.feed root@192.168.123.3:/etc/netwitness/ng/feed
scp *.feed root@192.168.123.2:/etc/netwitness/ng/feed
NwConsole -c "login 192.168.123.2:50004 admin netwitness" -c "/decoder/parsers feed op=notify" -c "exit"
NwConsole -c "login 192.168.123.3:50002 admin netwitness" -c "/decoder/parsers feed op=notify" -c "exit"
done
Notes
An example of the script running on my test system:
[root@rsareNsa ~]# ./ManualDeployFeeds.sh
/var/lib/netwitness/uax/scheduler/a5dc6f31-9924-4365-be5f-22e93b62b0f5
Websense.xml
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc. All Rights Reserved.
>feed create Websense.xml
Creating feed Websense...
done. 167 entries, 0 invalid records
All feeds complete.
>exit
Websense.feed 100% 7830 7.7KB/s 00:00
Websense.feed 100% 7830 7.7KB/s 00:00
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc. All Rights Reserved.
>login 192.168.123.2:50004 admin netwitness
Successfully logged in as session 128854
>/decoder/parsers feed op=notify
Success
>exit
(F) 2015-Aug-27 08:23:10 [ChannelManager::messageHandler] Socket Error: Operation canceled
Logged out of 192.168.123.2:50004
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc. All Rights Reserved.
>login 192.168.123.3:50002 admin netwitness
Successfully logged in as session 132591
>/decoder/parsers feed op=notify
Success
>exit
(F) 2015-Aug-27 08:23:10 [ChannelManager::messageHandler] Socket Error: Operation canceled
Logged out of 192.168.123.3:50002
/var/lib/netwitness/uax/scheduler/ca159f3e-b80d-4252-9a76-5573209fa3da
ECAT40.xml
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc. All Rights Reserved.
>feed create ECAT40.xml
Creating feed ECAT40...
done. 150 entries, 0 invalid records
All feeds complete.
>exit
ECAT40.feed 100% 4657 4.6KB/s 00:00
ECAT40.feed 100% 4657 4.6KB/s 00:00
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc. All Rights Reserved.
>login 192.168.123.2:50004 admin netwitness
Successfully logged in as session 128885
>/decoder/parsers feed op=notify
Success
>exit
(F) 2015-Aug-27 08:23:43 [ChannelManager::messageHandler] Socket Error: Operation canceled
Logged out of 192.168.123.2:50004
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc. All Rights Reserved.
>login 192.168.123.3:50002 admin netwitness
Successfully logged in as session 132622
>/decoder/parsers feed op=notify
Success
>exit
(F) 2015-Aug-27 08:23:43 [ChannelManager::messageHandler] Socket Error: Operation canceled
Logged out of 192.168.123.3:50002
/var/lib/netwitness/uax/scheduler/f84130cc-db1c-4bed-8c2a-3defca1f80a4
NetworkNamesCIDR.xml
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc. All Rights Reserved.
>feed create NetworkNamesCIDR.xml
Creating feed NetworkNamesCIDR...
done. 16 entries, 0 invalid records
All feeds complete.
>exit
NetworkNamesCIDR.feed 100% 732 0.7KB/s 00:00
NetworkNamesCIDR.feed 100% 732 0.7KB/s 00:00
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc. All Rights Reserved.
>login 192.168.123.2:50004 admin netwitness
Successfully logged in as session 128918
>/decoder/parsers feed op=notify
Success
>exit
(F) 2015-Aug-27 08:24:17 [ChannelManager::messageHandler] Socket Error: Operation canceled
Logged out of 192.168.123.2:50004
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc. All Rights Reserved.
>login 192.168.123.3:50002 admin netwitness
Successfully logged in as session 132654
>/decoder/parsers feed op=notify
Success
>exit
(F) 2015-Aug-27 08:24:17 [ChannelManager::messageHandler] Socket Error: Operation canceled
Logged out of 192.168.123.3:50002
/var/lib/netwitness/uax/scheduler/e3fadf43-0de7-4783-b576-482ac9b773f1
CollectionTypeFeed.xml
RSA Security Analytics Console 10.5.0.1.5599
Copyright 2001-2015, RSA Security Inc. All Rights Reserved.
>feed create CollectionTypeFeed.xml
Creating feed CollectionTypeFeed...
done. 9 entries, 0 invalid records
All feeds complete.
>exit
CollectionTypeFeed.feed 100% 391 0.4KB/s 00:00
CollectionTypeFeed.feed 100% 391 0.4KB/s 00:00
More information on creating a custom feed can be found in the Security Analytics documentation.
For instance in 10.5 this would be done using the instructions in the Security Analytics 10.5 User Guide.