Post upgrade Winrm collection stopping frequently due to Kerberos tickets in RSA Security Analytics 10.6.0.2

[#1]

Article Number

000033580

Applies To

RSA Product Set: Security Analytics
RSA Product/Service Type: SA Core Appliance
RSA Version/Condition: 10.6.0.2
 

Issue

Post upgrade, winrm collection stops periodically with 401 errors and failure message that contains below kerberos errors.

/var/log/messages:

Jul 14 12:17:52 XXX NwLogCollector[23687]: [WindowsCollection] [failure] [hostname] [processing] [WorkUnit] [processing] Unable to subscribe for events with Windows event source hostname: 401/Unauthorized.Possible causes:- Event source (hostname) does not map to a Kerberos Realm.
Jul 14 12:17:52 XXX NwLogCollector[23687]: [Krb5CacheMonitor] [failure] Failed to fetch Kerberos TGT for principal : username@DOMAIN.COM

- Use klist -A command to verify the tickets exist or not. 

sample output is as below if kerberos tickets exist.

[root@XXX ~]# klist -A
Ticket cache: DIR::/var/netwitness/logcollector/runtime/krb5_ccache_dir/tkthScmPw
Default principal: username@DOMAIN.COM

Valid starting     Expires            Service principal
07/19/16 04:52:19  07/19/16 14:52:19  krbtgt/DC@DOMAIN.COM
        renew until 07/19/16 14:52:19
07/19/16 04:52:25  07/19/16 14:52:19  HTTP/hostname@DOMAIN.COM
        renew until 07/19/16 14:52:19

- Running below two commands in Collector can regenerate the kerberos tickets and starts winrm collection automatically. But the winrm collection stops again when kerberos tickets were not refreshed again.

export KRB5CCNAME=DIR:/var/netwitness/logcollector/runtime/krb5_ccache_dir
kinit username@DOMAIN.COM

Cause

The kerberos tickets may not be refreshing due to either credentials in Event Category or krb5.conf file missing default realm.

Resolution

Please use below steps to resolve the issue permanently.

1. Login to Security Analytics GUI as admin.
2. Navigate to Collector->Config->Event Sources->Windows/Config page.
3. Select the Configuration of problematic Domain Controller in Event Categories page and click edit.
4. Re-enter the credentials as highlighted below and click OK.
Image description

5. Then login to putty session of collector and edit /etc/krb5.conf file to configure the default_realm entry for problematic domain under [libdefaults].

sample /etc/krb5.conf file contents are below.

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
default_realm = VCLOUD.LOCAL
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false

[realms]
VCLOUD.LOCAL = {
kdc = DC1.VCLOUD.LOCAL
admin_server = DC1.VCLOUD.LOCAL

[domain_realm]
.vcloud.local = VCLOUD.LOCAL
vcloud.local = VCLOUD.LOCAL

6. Run below two commands to regenerate kerberos tickets.

export KRB5CCNAME=DIR:/var/netwitness/logcollector/runtime/krb5_ccache_dir
kinit username@DOMAIN.COM

7. Verify the winrm logs by navigating to Investigation page in GUI.