Winrm 401/Unauthorized Does not Map to a Kerberos Realm

[david_waugh]

Hi we have many windows servers in our environment, but for one server I cannot get winrm to work.

1) I have run the winrmconfig.ps script on the server. The output is all BLUE/Green and Yellow. The user has been specified

04/12/2017 14:20:21
CURRENT LISTENER(S) INFORMATION:

04/12/2017 14:20:21 Listener: Address = * Transport = HTTPS Port = 5986 Hostname =MYSERVER.MYDOMAIN.COM Enabled = true URLPrefix = wsman CertificateThumbprint = 2e 05 19 d9 45 6f f8 3e 59 7f 23 10 81 fc e7 d16a 70 9d fb ListeningOn = 10.X.X.X, 10.X.X.X, 127.0.0.1, 172.X.X.X, ::1

04/12/2017 14:20:21 SECURITY LOG ACCESS FOR NETWORK SERVICE ACCOUNT CHECK BEGINS(WINRM SERVICE USES THIS ACCOUNT TO READ EVENT LOGS)
04/12/2017 14:20:21 Network Service SID is already added to the Security Channel ACL (Security Analytics can collect Security Event logs using the myuser@MYDOMAIN account)
04/12/2017 14:20:21 SECURITY LOG ACCESS FOR NETWORK SERVICE ACCOUNT CHECK ENDS
04/12/2017 14:20:21
COLLECTION USER RIGHTS CONFIGURATION BEGINS...

04/12/2017 14:20:21 Domain: BOE
04/12/2017 14:20:21 Account: MYUSER
04/12/2017 14:20:21 Checking access to the WinRM WMI Plugin (necessary for SID resolution)
04/12/2017 14:20:21 Winrm WMI plugin SDDL:
<PlugInConfiguration xmlns="http://schemas.microsoft.com/wbem/wsman/1/config/PluginConfiguration" Name="WMI Provider" Filename="C:\Windows\system32\WsmWmiPl.dll" SDKVersion="1" XmlRenderingType="text" UseSharedProcess="false" ProcessIdleTimeoutSec="0" RunAsUser="" RunAsPassword="" AutoRestart="false" Enabled="true" OutputBufferingMode="Block"><Resources><Resource ResourceUri="http://schemas.microsoft.com/wbem/wsman/1/wmi" SupportsOptions="true"><Security Uri="" ExactMatch="false" Sddl="O:NSG:BAD:P(A;;GA;;;BA)(A;;GA;;;IU)(A;;GA;;;RM)(A;;GR;;;S-1-5-21-606747145-527237240-6123456789-123346)S:P(AU;FA;GA;;;WD)(AU;SA;GWGX;;;WD)" /><Capability Type="Identify" /><Capability Type="Get" SupportsFragment="true" /><Capability Type="Put" SupportsFragment="true" /><Capability Type="Invoke" /><Capability Type="Create" /><Capability Type="Delete" /><Capability Type="Enumerate" SupportsFiltering="true" /><Capability Type="Subscribe" SupportsFiltering="true" /></Resource><Resource ResourceUri="http://schemas.dmtf.org/wbem/wscim/1/cim-schema" SupportsOptions="true"><Security Uri="" ExactMatch="false" Sddl="O:NSG:BAD:P(A;;GA;;;BA)(A;;GA;;;IU)(A;;GA;;;RM)S:P(AU;FA;GA;;;WD)(AU;SA;GWGX;;;WD)" /><Capability Type="Get" SupportsFragment="true" /><Capability Type="Put" SupportsFragment="true" /><Capability Type="Invoke" /><Capability Type="Create" /><Capability Type="Delete" /><Capability Type="Enumerate" /><Capability Type="Subscribe" SupportsFiltering="true" /></Resource><Resource ResourceUri="http://schemas.dmtf.org/wbem/wscim/1/*" SupportsOptions="true" ExactMatch="true"><Security Uri="" ExactMatch="false" Sddl="O:NSG:BAD:P(A;;GA;;;BA)(A;;GA;;;IU)(A;;GA;;;RM)S:P(AU;FA;GA;;;WD)(AU;SA;GWGX;;;WD)" /><Capability Type="Enumerate" SupportsFiltering="true" /><Capability Type="Subscribe" SupportsFiltering="true" /></Resource><Resource ResourceUri="http://schemas.dmtf.org/wbem/cim-xml/2/cim-schema/2/*" SupportsOptions="true" ExactMatch="true"><Security Uri="" ExactMatch="false" Sddl="O:NSG:BAD:P(A;;GA;;;BA)(A;;GA;;;IU)(A;;GA;;;RM)S:P(AU;FA;GA;;;WD)(AU;SA;GWGX;;;WD)" /><Capability Type="Get" SupportsFragment="false" /><Capability Type="Enumerate" SupportsFiltering="true" /></Resource></Resources><Quotas MaxConcurrentUsers="100" MaxConcurrentOperationsPerUser="100" MaxConcurrentOperations="1500" /></PlugInConfiguration>
04/12/2017 14:20:21 setWmiSDDL looking for SID: S-1-5-21-606747145-527237240-6123456789-123346
04/12/2017 14:20:21 setWmiSDDL SDDL: O:NSG:BAD:P(A;;GA;;;BA)(A;;GA;;;IU)(A;;GA;;;RM)(A;;GR;;;S-1-5-21-606747145-527237240-6123456789-123346)S:P(AU;FA;GA;;;WD)(AU;SA;GWGX;;;WD)
04/12/2017 14:20:21 User myuser@MYDOMAIN with SID S-1-5-21-606747145-527237240-6123456789-123346 is already
added to the WinRM WMI Plugin SDDL (Security analytics can resolve SIDs with this account)

04/12/2017 14:20:21 Checking access to the CIM Root (necessary for Event log collection)
04/12/2017 14:20:21 Checking access to the CIM Root for SID S-1-5-21-606747145-527237240-6123456789-123346
04/12/2017 14:20:21 Setup call to GetSecurityDescriptor, parameters:
04/12/2017 14:20:21 Got acl
04/12/2017 14:20:21 Found trustee SID: S-1-5-21-606747145-527237240-6123456789-123346

04/12/2017 14:20:21 User myuser@MYDOMAIN with SID: S-1-5-21-606747145-527237240-6123456789-123346 is already enabled
for WMI access via WinRM (Security Analytics can collect Event logs using this account)

04/12/2017 14:20:21 Checking user myuser@MYDOMAIN membership to the Event Log Readers group
04/12/2017 14:20:22 Event Log Readers member: NETWORK SERVICE
04/12/2017 14:20:22 Event Log Readers member: MYUSER
04/12/2017 14:20:22 User myuser@MYDOMAIN is already a member of Event Log Readers group
04/12/2017 14:20:22
COLLECTION USER RIGHTS CHECK ENDS HERE...

04/12/2017 14:20:22 OS Version:

2) From the logcollector I can curl to the Server on port 5986 (the https port successfully)

3) I have looked in the steps of the winrm troubleshooting guide and can see that I have kerberos tickets for this service.

Ticket cache: DIR::/var/netwitness/logcollector/runtime/krb5_ccache_dir/tktWWLHCf
Default principal: winrm@MYDOMAIN.COM

Valid starting Expires Service principal
12/04/17 11:48:45 12/04/17 21:48:54 krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
renew until 12/11/17 11:48:45

12/04/17 11:49:25 12/04/17 21:48:54 HTTP/myserver.mydomain.com@MYDOMAIN.COM
12/04/17 11:51:41 12/04/17 21:48:54 host/myserver.mydomain.com@MYDOMAIN.COM

4) I did a packet capture and can see that packets are flowing. But am wondering if there is an issue establishing the TLS connection

5) winrm get/winrm/config seems to be correct

6) I looked in the event log and saw the following

7) setspn -L hostname returns

C:\Windows\system32>setspn -L SERVER
Registered ServicePrincipalNames for CN=SERVER,OU=DEV,OU=Standard,OU=Member Servers 2012,DC=mydomain DC=com:
WSMAN/SERVER
WSMAN/SERVER.mydomain.com
TERMSRV/SERVER.mydomain.com
TERMSRV/SERVER
RestrictedKrbHost/SERVER
HOST/SERVER
RestrictedKrbHost/SERVER.mydomain.com
HOST/SERVER.mydomain.com

😎 The packet capture reveals that there is an encryption alert 21

9) Windows Log is showing:

Does anyone have any ideas?

[ConODonnell]

Have you looked into using a Managed Service Account fro that blackberry service (it seems on paper that SPN's are automatically handled) ???

[david_waugh]

I haven't and I don't have control over the Server. I'm thinking that the best way around this would be to forward the event logs to another server with the windows event log forwarding and then to collect the logs for this machine from the forwarded server.

Anyway hope you are well! It was good to meet you at techfest the other year.

[ConODonnell]

Always a pleasure Buddy !!!

Well i do have a case open with Microsoft, i would expect that at least if the (in this case blackberry) app could use a port spn (which in these cases ususally those spns are created when the app is using a non-standard HTTP port then it would work. I did some testing last night and had some mixed results on that so testing continues, as soon as i have a solidly reproducable environment i will turn it over to Microsoft.

[ConODonnell]

Some more on the topic and possible workaround:

When a WinRM Category is added for a domain (in the Collector’s event source config) the collector requests a TGT (Ticket To grant Tickets AKA Ticket Granting Ticket) from the configured Domain Controller (DC) using the collection user’s credentials. With the TGT acquired the collector issues a request to the Domain Controller to get a Service Ticket (ST) which will allow it HTTP access to the WinRM Service on the target system (the system we are attempting to collect from). The request contains a string known as a SPN (Service Principal Name) and typically takes the form HTTP/<fqdn of system> for example if the fqdn specified in the event source is mypc.testlab.com then the SPN will be HTTP/mypc.testlab.com where HTTP is known as the service type which for WinRM is HTTP as the underlying protocol in WinRM is HTTP.

Normally with SPN related issues an ST is acquired from the DC with no issues but when the collector presents that ST to the target system it is rejected with a krb_ap_err_modified error.

Typically an ST is rejected because the request cannot be decrypted which occurs if the token returned by the DC was not for the system.

Customer sees test connection is failing with: Test connection failed:Error! 401/Unauthorized. Possible causes: - Event source (mypc.testlab.com) does not map to a Kerberos Realm.

The error above is not necessarily always caused by SPN issues (I would check that the collection user belongs to the Event Log Readers group first, but if that is not the issue then

It’s very possibly SPN related and detecting SPN rejection issues is fairly easy process:

On the collector after doing the typical export command:

export KRB5CCNAME=DIR:/var/netwitness/logcollector/runtime/krb5_ccache_dir

issuing  a

klist -A

which should show the Kerberos cache contents:

Valid starting       Expires              Service principal

06/25/2018 17:40:35  06/26/2018 03:40:35  krbtgt/ TESTLAB.COM@TESTLAB.COM

        renew until 07/02/2018 17:40:35

06/25/2018 17:43:55  06/26/2018 03:40:35  HTTP/mypc@TESTLAB.COM

        renew until 07/02/2018 17:40:35

In the case above the cache dump shows an ST was acquired for our configured system (or in reality the SPN we requested one for)!

To really narrow it down we need to check for a Kerberos related error when the token is presented to the target system by the collector

tcpdump -w trace.pcap host mypc.TESTLAB.COM

Replacing mypc.TESTLAB.COM with the ip or fqdn of the target system. Loading the resulting pcap into wireshark should show the krb_ap_err_modified return from the system highlighted as an

easily locatable error while scrolling through the pcap. While the tcpdump is running click the test button and then do a ctrl-c to end the tcpdump command when the test connection fails.

If the krb_ap_err_modified response appears in the trace then the likely issue is that the token in the Kerberos cache the collector got from the DC was created for either:

1. Another system
2. A non-system resource i.e. an account

When the request for the ST reaches the DC it performs a ldap search of objects  in the domain which contains the attribute ServicePrincipalName for the SPN string in our request. If an object contains the attribute with that SPN then a token is created and returned for that object. For case 1 above it’s possible that the hostname keyed into the event source is wrong i.e. the user keys in

testpco1 instead of testpc01

but testpco1 then turns out to be a DNS tombstone i.e. possibly the system was renamed from testpco1 to t testpc01 so a record of the system’s old name still appears in DNS and in the DC. The DC then finds the entry for the HTTP/testpco1 SPN and returns a token for that system. When presented to testpc01 the token cannot be decrypted by that system since it was encrypted with a key for testpco1.

This issue is easy to diagnose, simply executing “hostname” in a DOS window on the failing system and closely comparing that to the hostname in the event source should reveal the difference.

The 2^nd failure case above is more common than the first i.e. it’s normal for customers to follow Microsoft’s guidelines to assign a system SPN (HTTP/mypc.testlab.com in our previous example) to a domain account.  This is required to give service account access to system resources across systems (common in Sharepoint web farms and a number of other clustered applications). The same process applies in that the DC does a search for the SPN string when the collector requests an ST for the target system in the domain but this time find’s it assigned to a domain account so it creates a token for that and returns it to the collector. The end result is the same in this case the token cannot be decrypted when submitted to the target system and so it gets rejected with the Kerberos error krb_ap_err_modified.

Once the DNS possibility is removed then It helps to search the domain for HTTP SPN’s assigned to resources other than the actual system they should be.

The windows setspn command can be used to search/add/modify SPNs for example:

setspn -Q HTTP/mypc*

will return all the SPNs in the domain that begin with mypc

seeing for example:

Checking domain DC=testlab,DC=com

CN=IISService,OU=Service Accounts,DC=testlab,DC=com

              HTTP/mypc.testlab.com

Should raise a red flag since the first CN= should ideally be CN=mypc

i.e. there should really only be ONE DN that contains the HTTP SPN we are asking for and that should be the DN of the actual system!

In this case:

Checking domain DC=testlab,DC=com

CN=mypc,OU=Domain Controllers,DC=testlab,DC=com

              HTTP/mypc.testlab.com

So what to do in these cases ???

The DNS issue is simple it’s purely a matter of a typo. The SPN issue is more complicated as you can actually delete the offending SPN and create it where it should be but that could break the your web farm!

If you look at a SPN it’s really a piece of text in other words collector sends a request to the DC with a string and if it finds that string assigned to a certain attribute then it returns a token for that object . The only stipulation for that is the string begins with HTTP/ to denote the class of service we want access to. As far as the DC is concerned the rest of the string does not really need to be an FQDN or hostname windows does not really care so if I assign a spn:

HTTP/oranges to mypc using setspn:

setspn -S HTTP/oranges mypc

then it would be accepted since im assigning the SPN to a valid object (mypc is a domain member server in this case).

The issue we would have here is that to get the collector to request this as a SPN from the DC then oranges would have to used as the hostname in the event source so the collector will fail in looking up the ip while trying to create a connection to connect to the target system.

One thing to try first is to just use the hostname as the event source rather than the fqdn (hostname will work for HTTP not HTTPS), again the SPN is just a string in this case the SPN would be HTTP/mypc

Using the setspn –Q you would check if there is a SPN of HTTP/mypc in the domain (and hopefully it’s assigned to the target System i.e.

setspn -Q HTTP/mypc*

Checking domain DC=testlab,DC=com

CN=mypc,OU=Domain Controllers,DC=testlab,DC=com

              HTTP/mypc

If this exact SPN does not exist in the domain then it can be created under the target system:

setspn -S HTTP/mypc mypc

But if the hostname based SPN exists under once again a different DN than that of the target system then a more drastic workaround may be necessary.

Creating a hard coded entry in the collector’s /etc/hosts that creates a “fake” hostname lookup, you can for example add an entry with the ip of the target system (192.168.12.2 in my case):

192.168.12.2    winrm-mypc

Then on the target system assign this SPN to the actual system:

setspn -S HTTP/winrm-mypc mypc

This solves the issue of getting a ST for this system by using a unique SPN while ensuring the collector can connect to the actual system as that unique string resolves to the system’s ip!

Happy SPN'ing.....