Logdecoder Capture stops intermittently due to less metadb size in RSA Security Analytics

[#1]

Article Number

000034156

Applies To

RSA Product Set: Security Analytics
RSA Product/Service Type: SA Core Appliance
RSA Version/Condition: 10.4.X, 10.5.X, 10.6.X
 

Issue

Logdecoder capture stops intermittently with below errors. 

/var/log/messages:
Sep 10 06:40:17 TCLDecDot NwLogDecoder[32074]: [Decoder] [warning] Meta database free space threshold exceeded (/var/netwitness/logdecoder/metadb, 2.65 GB free), capture is stopping.  Please check drive and configuration.
Sep 19 08:52:29 TCLDecDot NwLogDecoder[7353]: [Decoder] [warning] Meta database free space threshold exceeded (/var/netwitness/logdecoder/metadb, 2.92 GB free), capture is stopping.  Please check drive and configuration.
Sep 20 03:27:44 TCLDecDot NwLogDecoder[7353]: [Decoder] [warning] Meta database free space threshold exceeded (/var/netwitness/logdecoder/metadb, 2.99 GB free), capture is stopping.  Please check drive and configuration.
Sep 20 13:26:35 TCLDecDot NwLogDecoder[7353]: [Decoder] [warning] Meta database free space threshold exceeded (/var/netwitness/logdecoder/metadb, 2.69 GB free), capture is stopping.  Please check drive and configuration.

In this instance, the meta.free.space.min value in Logdecoder->Explore->Database->Config page is 3 GB

Cause

The usage of metadb/sessiondb/packetdb/indexdb which even if grows beyond the configured size is a normal scenario as long as rollover is occurring automatically before the filesystem fills, it is functioning as designed.

As rollover is not that precise and rollover is only active once the usage exceeds the specified size threshold, and only is activated periodically, rather than instantaneously.

So,it seems rollover starts periodically and in that mean time db grows more than 95%. This causes the free space available for core database directories getting reduced. But, the core services work when minimum required free space available.

Resolution

In this circumstance, The metadb should have minimum 3 GB free space to work logdecoder service. The log errors show capture stopped details when meta free space reduced to ❤️ GB. So, the log pattern says 2 GB meta free space setting is a good idea, Since the free space never reduced <2 GB.

Please follow below steps to solve this issue permanently.

1. Login to GUI and Navigate to Logdecoder->Explore view.
2. Left hand side expand database->config
3. Chang meta.free.space.min value from 3 GB to 2 GB.
 
This change would take effect immediately.