NetWitness Community

ThomasJones1 · ‎2019-07-08

Updated for snmpv3: 01/14/2020

Updated for snmpv3: 06/01/2020

Updated for snmpv1,2: 08/10/2020

Scenario –
You or your customer would like to link SNMP to the Netwitness for system monitoring purposes (Solarwinds, Nagios, etc.).

If you are planning on using snmpv3 and have already configured the iptables - please go the the version 3

Why SNMP?
SNMP is an “agentless” method of monitoring network devices and servers, which can be viable alternative to the problems, hassle, and maintenance associated with agents.

The process:

Preparation
1. How many hosts
2. Host locations and subnets
3. ssh access
4. Documents
  1. https://community.rsa.com/docs/DOC-93651
  2. https://community.rsa.com/docs/DOC-45725
Obtain the snmp script for netwitness (there may be other versions available)
1. nwsnmpconfig-2015.09.10.sh
Considerations
1. Change requests?
2. Schedules
3. Implementation
4. Manual configuration or script
5. Backout process
6. Expectations from the other stake holders
Backup
1. Netwitness.json (/etc/netwitness/config-management/environments/netwitness.json)
2. Iptables-config (/etc/sysconfig/iptables-config)
3. Iptables (/etc/sysconfig/iptables)
4. SNMP (/etc/snmp/snmpd.conf)
Implementation – Short version
1. Each host will need the netwitness services stopped before the system is configured for SNMP.
2. If netwitness is already deployed and the firewall rules during the install were not set to custom the netwitness.json file will need to be modified so that future updates and upgrades do not change the firewall rules.
3. Modify the iptables-config (configuration for firewall rules)
4. Modify the iptables files (firewall rules)
5. SNMP default port 161
6. Optional – it is also a good time to add icmp if desired.
7. Run the nwsnmpconfig-2015.09.10.sh script
8. Modify the snmp.conf file
9. Restart the firewall service
10. Restart the snmp service
11. Restart the netwitness services
Testing
1. Snmpwalk command is the best way to test.
2. First, run the snmpwalk on the current host
3. Second, use the snmpwalk from a different host

Sounds pretty easy, right? There is a lot going on with this process, so here is the detailed break down.

Backup
1. Netwitness.json (/etc/netwitness/config-management/environments/netwitness.json)
2. Iptables-config (/etc/sysconfig/iptables-config)
3. Iptables (/etc/sysconfig/iptables)
4. SNMP (/etc/snmp/snmpd.conf - if exists)
Stop all the services for netwitness (choose one of these host types - Network only - the basics). Possible host types: Archiver, Broker, Concentrator, Decoder, EndpointHybrid, EndpointLogHybrid, ESAPrimary, ESASecondary, LogCollector, LogDecoder, LogHybrid, Malware, NetworkHybrid, UEBA. Please reference the community docs for additional appliances.
1. SA Server
  1. systemctl stop jetty.service
  2. systemctl stop nwappliance.service
  3. systemctl stop nwbroker.service
  4. systemctl stop rsa-nw-admin-server.service
  5. systemctl stop rsa-nw-config-server.service
  6. systemctl stop rsa-nw-content-server.service
  7. systemctl stop rsa-nw-integration-server.service
  8. systemctl stop rsa-nw-investigate-server.service
  9. systemctl stop rsa-nw-orchestration-server.service
  10. systemctl stop rsa-nw-respond-server.service
  11. systemctl stop rsa-nw-security-server.service
  12. systemctl stop rsa-nw-source-server.service
2. Broker
  1. systemctl stop nwappliance.service
  2. systemctl is-active nwbroker.service
3. Decoder
  1. systemctl stop nwappliance.service
  2. systemctl stop nwdecoder.service
4. Concentrator
  1. systemctl stop nwappliance.service
  2. systemctl stop nwconcentrator.service
5. ESA Primary
  1. systemctl stop rsa-nw-contexthub-server.service
  2. systemctl stop rsa-nw-esa-analytics-server.service
  3. systemctl stop rsa-nw-esa-server.service
6. ESA Secondary
  1. systemctl stop rsa-nw-esa-analytics-server.service
  2. systemctl stop rsa-nw-esa-server.service
Update the files
1. Netwitness.json - CAUTION if put on any host other than the SA Server - upgrade will be halted because the hosts do not know what to do with it.
  1. "global" : {
    "customer-firewall" : true,
    "mongo" : {
2. Iptables-config
  1. IPTABLES_SAVE_ON_STOP="no" to IPTABLES_SAVE_ON_STOP="yes"
  2. IPTABLES_SAVE_ON_RESTART="no" to IPTABLES_SAVE_ON_RESTART="yes"
3. Iptables (make sure the rules are put at the top and not the bottom)
  1. OPEN the SNMP default port 161
  2. A INPUT -p udp -m udp --dport 161 -j ACCEPT
  3. Optional – Open icmp if the environment relies on ping for management purposes
  4. -A INPUT -p icmp -j ACCEPT
4. Set up SNMP
  1. chkconfig snmpd on
  2. Run the following script or later version
    1. nwsnmpconfig-2015.09.10.sh
    2. Don’t forget to change the permissions
  3. Snmpd.conf
    1. Uncomment and/or modify
    2. (uncomment) # master agentx to master agentx
    3. rocommunity netwitness
    4. (uncomment and add <ip.addr>) # agentaddress 192.x.x.x
      1. this is typically the ip address associated eth0 (bonded) or em1 on netwitness hosts
    5. trapcommunity netwitness
      1. this is normally the trap community the designation you obtain from an administrator – ex.ABC#$%123
5. Restart all the services
  1. systemctl restart snmpd.service
  2. systemctl restart iptables.service
  3. Restart the appliance Netwitness services
6. Testing
  1. Run the snmpwalk locally
    1. snmpwalk -v2c -Of -c netwitness -m "/usr/share/snmp/mibs/NETWITNESS-MIB.txt" 127.0.0.1 .1.3.6.1.4.1.36807
    2. snmpwalk -v 2c -c "communityName" x.x.x.x
  2. Run the snmpwalk from a different server.
    1. snmpwalk -v2c -Of -c netwitness -m "/usr/share/snmp/mibs/NETWITNESS-MIB.txt" <ip address of the queried nw host> .1.3.6.1.4.1.36807
  3. Have the administrator snmpwalk test
    i. snmpwalk -v 2c -c "communityName" x.x.x.x

SNMPv3 - authPriv (authentication + encryption)

Two files to be concerned with
1. /etc/snmp/snmpd.conf
2. /var/lib/net-snmp/snmpd.conf (Do not modify - just look at it)
Disable SNMPv1 and 2
1. /etc/snmp/snmpd.conf
2. service snmpd stop
Run the following command and put in your info.
1. For fips we must use SHA and AES
  1. I tried md5 + DES and it did not work.
2. net-snmp-create-v3-user -A "authPassphrase" -X "privacyPassphrase" -a SHA -x AES snmpUser
3. for secure snmp - authPriv you need two passphrases
Make sure file has been created and check the permissions on the file
1. /var/lib/net-snmp/snmpd.conf
  1. No need to modify this file. Just make sure the net-snmp-create-v3-user command modified the file.
2. service snmpd start
3. Test it with the following command. (Recommend testing from a different computer that is able to reach the host on port 161
  1. snmpwalk -v 3 -l authPriv -a SHA -A authPassphrase -x AES -X privacyPassphrase -u snmpUser 10.x.x.x

If you are doing a larger deployment, I would highly recommend scripting the process and as a best practice do your development in a test environment.

How did I do this in my most recent deployment?

Customer was on 11.2.0.1 with network only. 32 Hosts.

I started by referencing the documents above and the web for the script outline. I then scripted out the process, loaded the scripts on the SA head (MyScript and nwsnmpconfig-2015.09.10.sh). I pushed them out with the salt-cp '*' file.copy command to all the hosts. I then changed the scripts permissions with the salt '*' cmd.run 'chmod...'. At this point, I had put the files on all 32 hosts and ready to be run. Because I wanted to ensure there were no problems, I ssh'd into all the hosts and ran the scripts manually. Lastly, I would advise being careful with the salt '*' command because that updates all the hosts at once. Alternately, the salt command can be used targeting one host by using the salt minion id. Ex. salt '32b78ed8-a0dc-4f4c-915f-ec14aeacf6hf' cmd.run 'enter your command here'

Moving forward as we add additional hosts, I will simply WinSCP the scripts on to the new host, change the permissions, and run the scripts.

Let me know your thoughts - Tom J

KapitanAl · ‎2020-07-30

Thomas Jones‌,

I tested on fresh deployment (OVA) of NetWitness 11.4.

Article 000036446 (https://community.rsa.com/docs/DOC-93651) describes adding the custom iptables rules in a way that differs from yours:
+ /etc/sysconfig/iptables-config file is not modified;
+ iptables service is reloaded after /etc/sysconfig/iptables is modified, not restarted;
+ change in /etc/netwitness/config-management/environments/netwitness.json is to be applied on every host on which custom firewalls rules are needed and persistent between updates while you imply to have/modify netwitness.json only on SA Server ("Netwitness.json - CAUTION if put on any host other than the SA Server - upgrade will be halted because the hosts do not know what to do with it.").
Moreover following your instructions custom iptables rules from the freshly modified /etc/sysconfig/iptables will not be utilized - issuing systemctl restart iptables.service after setting IPTABLES_SAVE_ON_RESTART="yes"/IPTABLES_SAVE_ON_STOP="yes" will overwrite the /etc/sysconfig/iptables with the currently used/operating rules.

Following the steps from article 000036446 (https://community.rsa.com/docs/DOC-93651) allows iptables custom rules to persist across host reboots.

Reading the article 000037864 (https://community.rsa.com/docs/DOC-106839) makes it even more confusing ("In Netwitness 11.X appliances, editing /etc/sysconfig/iptables file using vi editor will not hold the changes permanently even after applying Netwitness.json settings").

iptables rules I created in accordance with the syntax of already present (out-of-the-box):
-A INPUT -p icmp -m icmp --icmp-type 8 -m comment --comment "ICMP Echo Requests" -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -p udp -m udp -m multiport --dports 161 -m comment --comment "SNMP Queries" -m conntrack --ctstate NEW -j ACCEPT
ICMP rule allows only incoming Echo Request (ping) not the whole ICMP communication.

NETWITNESS-MIB.txt is located at /var/snmp/mibs/NETWITNESS-MIB.txt not /usr/share/snmp/mibs/NETWITNESS-MIB.txt (same as NETWITNESS-IPMI-MIB.txt).

MIB looks to be incomplete because in output of command snmpwalk -v2c -Of -c netwitness -m "/var/snmp/mibs/NETWITNESS-MIB.txt" 127.0.0.1 .1.3.6.1.4.1.36807 numerical OIDs are present:
(...)
.iso.org.dod.internet.private.enterprises.netwitness.nwProducts.13.1.1.3.313 = STRING: "Linux 3.10.0-1062.4.1.el7.x86_64 #1 SMP Fri Oct 18 17:15:30 UTC 2019 x86_64"
.iso.org.dod.internet.private.enterprises.netwitness.nwProducts.13.1.1.3.314 = STRING: "2048,34 minutes 8 seconds"
.iso.org.dod.internet.private.enterprises.netwitness.nwProducts.13.1.1.3.315 = STRING: "f97e8416-f537-4255-8837-f402f4ab3bc5"
.iso.org.dod.internet.private.enterprises.netwitness.nwProducts.13.1.1.3.316 = STRING: "11.4.0.0-14847.5.6a1ea2953"

Also I have added the -ro option into the SNMPv3 user creation command to give it read-only permissions (net-snmp-create-v3-user -ro -A "authPassphrase" -X "privacyPassphrase" -a SHA -x AES snmpUser).

After doing all that I found article number 000038677 (https://community.rsa.com/docs/DOC-111886) which describes configuring SNMPv3 but without running the nwsnmpconfig-2015.09.10.sh...

ThomasJones1 · ‎2020-08-10

Albert,

Thank you so much for your feed back. It has been a year or two since I developed this process and I know other documents have emerged. That said, this is just an example of how I configured a single production environment. As always, I recommend that you defer to the manual and customer support documents as they are the sources of record. As you have noticed many of our documents remain disjointed. This can be problematic at times but rest assured, the document team is always working on making this site easier for customers to navigate and use the content. Hopefully, my write up helped a little.

KapitanAl · ‎2020-08-11

Thomas Jones‌,

You are welcome.

I am currently working with Customer Support/Engineering on other aspects related to SNMP so will be adding the resolutions here for the wider audience to use.

KapitanAl · ‎2020-09-02

Confirmed by Engineering through Customer Support - no SNMP monitoring for ESA, MIB file lacks definitions of "nwLogCollector / OBJECT IDENTIFIER ::= {nwProducts 13} " OID branch (however service itself is providing the data). Maybe it will be added in future release.

Core Services need to be restarted after every Host reboot for SNMP to operate - waiting for explanation why and workaround.

Room for improvement is out there

BrianThompson2 · ‎2020-09-17

Albert Cieszkowski wrote:

Confirmed by Engineering through Customer Support - no SNMP monitoring for ESA, MIB file lacks definitions of "nwLogCollector / OBJECT IDENTIFIER ::= {nwProducts 13} " OID branch (however service itself is providing the data). Maybe it will be added in future release.

Core Services need to be restarted after every Host reboot for SNMP to operate - waiting for explanation why and workaround.

Room for improvement is out there

Is this still true?
"Core Services need to be restarted after every Host reboot for SNMP to operate"

KapitanAl · ‎2020-09-23

Brian Thompson‌,

According to Customer Support - yes.

Case ID is: 01639521; Engineering Ticket ID is: ASOC-103163.

Should be resolved in future release, although ETA is unknown.

NetWitness Community

SNMP with Netwitness Appliances - SNMPv1,2 and 3 – Put it all together 11.x