Article Number
000031300
Applies To
RSA Product Set: Security Analytics, NetWitness Logs & Network
RSA Product/Service Type: Concentrator, Log Decoder
Platform: CentOS 6, CentOS 7
RSA Product/Service Type: Concentrator, Log Decoder
Platform: CentOS 6, CentOS 7
Issue
Each meta key that is indexed in the RSA NetWitness Platform has a valueMax value associated with it. This is the maximum number of unique values that can be stored in the index for this meta key.
This is defined in the index-concentrator.xml and index-concentrator-custom.xml values on a Security Analytics concentrator.
For example:
If this value is exceeded, then the additional unique values will not be able to be seen when investigating, although it will still be recorded in the session information.
It is therefore important to be alerted if the index for the meta key is full.
This is defined in the index-concentrator.xml and index-concentrator-custom.xml values on a Security Analytics concentrator.
For example:
<key description="Hostname Aliases" level="IndexValues" name="alias.host" format="Text" valueMax="2500000" />If this value is exceeded, then the additional unique values will not be able to be seen when investigating, although it will still be recorded in the session information.
It is therefore important to be alerted if the index for the meta key is full.
In some cases, the number of unique values for a key may exceed any setting of ValueMax that is used. For example, if you were to index URLs seen by the system then the index for this meta key would become quickly full due to the large number of unique possible values. For source ports and destination ports for a TCP session then there is a maximum of 65536 possible values so the valueMax is set to this value.
When a meta key is full the following will be seen in /var/log/messages on the concentrator:
Sep 18 16:40:20 logconc nw[11922]: [Index] [warning] Index key alias.host has reached max capacity of 2500000 values and will ignore new values for this slice.Task
We wish to be alerted if a metakey becomes full so that we can plan accordingly. To do this
1. Install the RSA Security Analytics Parsers from RSA Link here: https://community.rsa.com/docs/DOC-28866
2. Add the following message to the RSA Security Analytics parser above the </DEVICEMESSAGES> tag.
4. Add the following to your Log Decoder at /etc/netwitness/ng/envision/etc/table-map-custom.xml
5. When the Index Key becomes full the event description "Index Key Full" will become populated.
Image description
1. Install the RSA Security Analytics Parsers from RSA Link here: https://community.rsa.com/docs/DOC-28866
2. Add the following message to the RSA Security Analytics parser above the </DEVICEMESSAGES> tag.
<MESSAGE
level="1"
parse="1"
parsedefvalue="1"
tableid="1"
id1="Index:45"
id2="Index"
eventcategory="1612010000"
content="<@event_description:Index Key Full>[warning] Index key <metakey> has reached max capacity of <fld2> values and will ignore new values for this slice."/>
<key description="Meta Keys" level="IndexValues" name="metakey" format="Text" valueMax="1000" />4. Add the following to your Log Decoder at /etc/netwitness/ng/envision/etc/table-map-custom.xml
<mapping envisionName="metakey" nwName="metakey" flags="None" format="Text"/>5. When the Index Key becomes full the event description "Index Key Full" will become populated.
Image descriptionResolution
If an index key does become full then there are several options.
Attached to this article is a pair of scripts to help report on your current index slices. Please note the index-profile.pl script is meant for 10.X releases while the index-profile.py is meant for 11.X releases. Please download and follow the instructions of the one that best fits your environment.
For Customers using version 10.X:
Please use the index-profile-10.X.zip file. This contains a perl script.
Usage: ./index-profile.pl concentrator_ip [out_file_name]
For example, to take an index profile snapshot every 30 min create a cron job for every concentrator as follows:
This will generate a CSV file which then can be easily analyzed, so the optimal values of ValueMax, save.session.count or the index save scheduler can be worked out.
The sample output:
For Customers using the 11.X release:
Please use the index-profile-11.X.zip file. This contains a python script.
The python version of this script is meant for 11.X and has all the same functionality as the first script as well as a couple more options. This script requires Python 2.7.14 and can work either on either the Concentrator device itself or on a Windows or Mac workstation with Python 2.7 installed. All of the options can be found by running the command with the "-h" option. Here is a sample output of what that would look like:
A couple things to note about this script:
You can do a similar thing with a cron job as in 10.X but like so:
- Create more index slices
- Increase the valueMax value for the IndexKey
- Switch off indexing for the key
- Do nothing - it may not be practical to capture all unique values for the key or it may be that all possible unique values have been captured.
Attached to this article is a pair of scripts to help report on your current index slices. Please note the index-profile.pl script is meant for 10.X releases while the index-profile.py is meant for 11.X releases. Please download and follow the instructions of the one that best fits your environment.
For Customers using version 10.X:
Please use the index-profile-10.X.zip file. This contains a perl script.
Usage: ./index-profile.pl concentrator_ip [out_file_name]
For example, to take an index profile snapshot every 30 min create a cron job for every concentrator as follows:
# Concentrators 1
0,30 * * * * /root/index-profile-1.2.pl <Concentrator1_IP> index-snap-1.csv >> index-snap-1.csv
# Concentrator 2
0,30 * * * * /root/index-profile-1.2.pl <Concentrator2_IP> index-snap-2.csv >> index-snap-2.csvThis will generate a CSV file which then can be easily analyzed, so the optimal values of ValueMax, save.session.count or the index save scheduler can be worked out.
The sample output:
| Session | sourcefile | sinterface | content | policy.name | device.name | tld | city.src | alias.host | category | udp.srcport | event.computer | country.src | |
| 9472259 | 0.00% | 0.00% | 0.00% | 4.45% | 0.00% | 0.00% | 0.00% | 4.65% | 2.81% | 5.87% | 0.00% | 0.10% | 2.13% |
| 26421267 | 0.00% | 0.00% | 0.00% | 9.58% | 0.00% | 0.00% | 0.00% | 6.70% | 5.43% | 7.23% | 0.00% | 0.10% | 2.38% |
| 18719236 | 0.00% | 0.00% | 0.00% | 8.11% | 0.00% | 0.00% | 0.00% | 6.30% | 4.20% | 6.66% | 0.00% | 0.10% | 2.40% |
| 33624165 | 0.00% | 0.00% | 0.00% | 13.62% | 0.00% | 0.00% | 0.00% | 7.69% | 6.38% | 7.70% | 0.00% | 0.10% | 2.55% |
| 48796236 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| 62908505 | 0.00% | 0.00% | 0.00% | 32.48% | 0.00% | 0.00% | 0.00% | 9.54% | 9.91% | 8.94% | 0.00% | 0.10% | 2.71% |
| 76682159 | 0.00% | 0.00% | 0.00% | 40.05% | 0.00% | 0.00% | 0.00% | 10.34% | 11.36% | 9.39% | 0.00% | 0.10% | 2.78% |
| 88894056 | 0.00% | 0.00% | 0.00% | 44.02% | 0.00% | 0.00% | 0.00% | 11.11% | 12.70% | 9.73% | 0.00% | 0.10% | 2.81% |
| 100426583 | 0.00% | 0.00% | 0.00% | 50.97% | 0.00% | 0.00% | 0.00% | 11.59% | 13.84% | 10.04% | 0.00% | 0.10% | 2.84% |
| 115804180 | 0.00% | 0.00% | 0.00% | 57.16% | 0.00% | 0.00% | 0.00% | 12.19% | 15.27% | 10.38% | 0.00% | 0.10% | 2.86% |
| 129613715 | 0.00% | 0.00% | 0.00% | 65.50% | 0.00% | 0.00% | 0.00% | 12.68% | 16.49% | 10.65% | 0.00% | 0.10% | 2.89% |
| 141790056 | 0.00% | 0.00% | 0.00% | 68.83% | 0.00% | 0.00% | 0.00% | 13.14% | 17.69% | 10.95% | 0.00% | 0.10% | 2.93% |
| 153489616 | 0.00% | 0.00% | 0.00% | 72.53% | 0.00% | 0.00% | 0.00% | 13.53% | 18.86% | 11.12% | 0.00% | 0.10% | 2.94% |
| 161713547 | 0.00% | 0.00% | 0.00% | 74.81% | 0.00% | 0.00% | 0.00% | 13.77% | 19.73% | 11.19% | 0.00% | 0.10% | 2.95% |
| 174303022 | 0.00% | 0.00% | 0.00% | 78.22% | 0.00% | 0.00% | 0.00% | 14.38% | 20.92% | 11.37% | 0.00% | 0.10% | 2.97% |
| 183091639 | 0.00% | 0.00% | 0.00% | 81.39% | 0.00% | 0.00% | 0.00% | 14.73% | 21.72% | 11.49% | 0.00% | 0.10% | 3.00% |
| 192322898 | 0.00% | 0.00% | 0.00% | 84.70% | 0.00% | 0.00% | 0.00% | 14.93% | 22.51% | 11.64% | 0.00% | 0.10% | 3.01% |
| 198379858 | 0.00% | 0.00% | 0.00% | 86.79% | 0.00% | 0.00% | 0.00% | 15.10% | 23.00% | 11.82% | 0.00% | 0.10% | 3.01% |
| 203343360 | 0.00% | 0.00% | 0.00% | 89.23% | 0.00% | 0.00% | 0.00% | 15.23% | 23.46% | 11.87% | 0.00% | 0.10% | 3.02% |
| 216383778 | 0.00% | 0.00% | 0.00% | 93.93% | 0.00% | 0.00% | 0.00% | 15.61% | 24.51% | 12.09% | 0.00% | 0.10% | 3.08% |
| 224762659 | 0.00% | 0.00% | 0.00% | 97.18% | 0.00% | 0.00% | 0.00% | 15.83% | 25.20% | 12.27% | 0.00% | 0.10% | 3.09% |
| 233054034 | 0.00% | 0.00% | 0.00% | 100.00% | 0.00% | 0.00% | 0.00% | 16.05% | 25.91% | 12.49% | 0.00% | 0.10% | 3.09% |
Please use the index-profile-11.X.zip file. This contains a python script.
The python version of this script is meant for 11.X and has all the same functionality as the first script as well as a couple more options. This script requires Python 2.7.14 and can work either on either the Concentrator device itself or on a Windows or Mac workstation with Python 2.7 installed. All of the options can be found by running the command with the "-h" option. Here is a sample output of what that would look like:
[root@localhost] ➤ ./index-profile.py -h
usage: index-profile.py [-h] [-f OUTPUTFILE] [--host HOST] [--port PORT] [-S]
[-u USERNAME] [-p PASSWORD] [--horizontal]
[--addHeaders] [-d]
Evaluate the status of the current index slices on the designated device. This
script returns a csv output/file that contains a listing on keys with their
undying usage of the most recent index slice they exist in. Note that if a key
is seldomly used, this may not be from the current slice. This value is
displayed as a percentage by default. The first number is the number of
sessions currently in this index slice. Think of this as the bookmark for our
results.
optional arguments:
-h, --help show this help message and exit
-f OUTPUTFILE, --OutputFile OUTPUTFILE
Designate an output filename that will be a CSV.
(indexstatus.csv by default)
--host HOST Define the host that we will check the index on.
(localhost by default)
--port PORT Define the port that we will attempt to connect on the
host. (50105 by default)
-S, -s, --SSL Enable this if the REST Port is using SSL. (False by
default)
-u USERNAME, --username USERNAME
Define the user that will connect on the Service API
(admin by default)
-p PASSWORD, --password PASSWORD
Define the password the user will use to connect to
the Service API (netwitness by default)
--horizontal Horizontal Output with raw values. Vertical/Default
output only puts percentages.
--addHeaders When appending an existing file, add the headers
before printing the next set of results again. This is
not necessary if the file is brand new
-d, -D Debug Mode: This really just helps out with what was collected when parsed from API.A couple things to note about this script:
- The output is similar to the one you see from the 10.X Perl script.
- A file is generated by default as well as standard output being generated. By default, this file is called "indexstatus.csv" and is written to the same directory the script is being executed from. The -f option can specify where this file is generated.
- Please note that if your REST interface is using HTTPS, you must pass the -S option into the script. You can navigate to http://concentrator_ip:50105 or https://concentrator_ip:50105 respectively if you'd like to know which one you are currently using.
- The password you use to login to admin in the UI may not be the same as the local service user account. Typically, this is admin/netwitness.
- By default, if the file exists, the header information will not be appended a second time unless specified by the --addHeaders parameter.
- There is a horizontal output option which will print output for each key in the following order if you want to see raw numbers: keyname,number of values occupying the last index file, valuemax, percentage used. Note: Horizontal is not compatible with appendHeaders.
You can do a similar thing with a cron job as in 10.X but like so:
# Concentrators 1 which has a non standard service username and password
0,30 * * * * /root/index-profile.py --host 192.168.2.101 -u admin -p NETWITNESS11 -f /root/index-status-1.csv > /dev/null 2>&1
# Concentrator 2 which has an SSL REST API enabled
0,30 * * * * /root/index-profile.py --host 192.168.2.102 -S -f /root/index-status-2.csv > /dev/null 2>&1Notes
All screenshots and hostnames are from an internal lab system.
In this article
