Archiver in initialization state and aggregation is not starting in RSA NetWitness Logs and Network

[#1]

Article Number

000002003

Applies To

RSA Product Set: NetWitness Logs and Network?
RSA Product/Service Type: Archiver
RSA Version/Condition: 10.6.X
 

Issue

Archiver service in initialization state and aggregation is not starting.
Further checking, Archiver service appears to have an issue with one of its collections - in this example 'ACVR11WINDOWS'
 

[root@CHEETAHACVR11 ACVR11WINDOWS]# tail -n10000 /var/log/messages | grep NwArchiver | grep fail 
May 31 16:04:43 CHEETAHACVR11 NwArchiver[25890]: [Engine] [failure] Module archiver failed to load: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker 
May 31 16:04:43 CHEETAHACVR11 NwArchiver[25890]: [Engine] [failure] Module archiver failed to load: Diagnostic information: Throw in function virtual nw::AggStatePtr nw::CollectionBroker::addAggState(nw::AggStatePtr, bool)Dynamic exception type: boost::exception_detail::clone_impl<nw::Exception>std::exception::what: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker[boost::errinfo_at_line_*] = 99 
May 31 16:04:45 CHEETAHACVR11 NwArchiver[25890]: [Broker] [failure] Internal collection {ACVR11WINDOWS} is in a failed state 
May 31 16:04:45 CHEETAHACVR11 NwArchiver[25890]: [Broker] [failure] Throw in function virtual void nw::CollectionBroker::online()Dynamic exception type: boost::exception_detail::clone_impl<nw::Exception>std::exception::what: Internal collection {ACVR11WINDOWS} is in a failed state[boost::errinfo_at_line_*] = 123 
Jun 1 03:44:03 CHEETAHACVR11 collectd[3610]: NgNativeReader_NwArchiver-FastUpdate: failed to connect to nws://admin@localhost:56008/?group=Administrators&cert=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fcerts%2Fed1d6d0e-7e90-487a-a3f6-95468cb2642b.pem&key=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fprivate_keys%2Fed1d6d0e-7e90-487a-a3f6-95468cb2642b.pem 
Jun 1 03:44:03 CHEETAHACVR11 NwArchiver[34313]: [Engine] [failure] SNMP AgentX Master connection is DOWN due to: No such file or directory. Likely cause: snmpd is disabled or misconfigured. 
Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [{ACVR11WINDOWS}] [failure] There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct 
Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [{ACVR11WINDOWS}] [failure] Throw in function virtual void nw::BrokerState::load()Dynamic exception type: boost::exception_detail::clone_impl<nw::LogicError>std::exception::what: There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct[boost::errinfo_at_line_*] = 1134 
Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [Aggregation] [failure] Failed to initialize device '{ACVR11WINDOWS}' because There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct. Device aggregation is being stopped. 
Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Engine] [failure] Module archiver failed to load: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker 
Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Engine] [failure] Module archiver failed to load: Diagnostic information: Throw in function virtual nw::AggStatePtr nw::CollectionBroker::addAggState(nw::AggStatePtr, bool)Dynamic exception type: boost::exception_detail::clone_impl<nw::Exception>std::exception::what: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker[boost::errinfo_at_line_*] = 99 
Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Broker] [failure] Internal collection {ACVR11WINDOWS} is in a failed state 
Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Broker] [failure] Throw in function virtual void nw::CollectionBroker::onlin 
Jun 1 08:03:49 CHEETAHACVR11 NwArchiver[11966]: [{ACVR11WINDOWS}] [failure] There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct

 

Cause

As seen from the error messages above, there seems to be some mappings problem on the collection;
 

Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [{ACVR11WINDOWS}] [failure] Throw in function virtual void nw::BrokerState::load()Dynamic exception type: boost::exception_detail::clone_impl<nw::LogicError>std::exception::what: There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct[boost::errinfo_at_line_*] = 1134 
Jun 1 03:44:12 CHEETAHACVR11 NwArchiver[34313]: [Aggregation] [failure] Failed to initialize device '{ACVR11WINDOWS}' because There is a mismatch between vSessions (452103) and lSessions (452102), please delete and re-add this device to correct. Device aggregation is being stopped.

Looks like mappings between lSession and vSession corrupted when root user initiated force shutdown by command and update could not persist. Archiver session will usually detect these errors and does not allow aggregation to start until they are fixed. 

Resolution

1. Determine the problematic collection, here it is ACVR11WINDOWS.

Jun 1 03:44:23 CHEETAHACVR11 NwArchiver[34313]: [Engine] [failure] Module archiver failed to load: Internal collection {ACVR11WINDOWS} has a failed status in the internal broker

2. SSH to Archiver service and shut down Archiver service. 

[root@Archiver ~]# systemctl stop nwarchiver

3. Navigate to the Archiver's index folder, usually /var/netwitness/archiver/index. and NOT specific collection's index folder (such as /var/netwitness/archiver/ACVR11WINDOWS/index)

4. Move the collection's index/mapping files to another directory ex. /tmp/windows/

mv ACVR11WINDOWS-* /tmp/windows/

5. Start Archiver service

[root@Archiver ~]# systemctl start nwarchiver