Introduction
This blog post demonstrates a common method as to how organisations can get compromised. Initially, the viewpoint will be from the attacker’s perspective, it will then move on to show what artifacts are left over within the RSA NetWitness Packets and RSA NetWitness Endpoint solutions that analysts could use to detect this type of activity.
Scenario
Apache Tomcat server exposed to the internet with weak credentials to the Tomcat Manager App gets exploited by an attacker. The attacker uploads three Web Shells, confirms access to all of them and then uploads Mimikatz to dump credentials.
Definitions
Web Shells
A web shell is a script that can be uploaded to a web server to enable remote administration of the machine. Infected web servers can be either internet-facing or internal to the network, where the web shell is used to pivot further to internal hosts.
A web shell can be written in any language that the target web server supports. The most commonly observed web shells are written in languages that are widely supported, such as JSP, PHP, ASP, Perl, Ruby, Python, and Unix shell scripts are also used.
Mimikatz
Mimikatz is an open source credential dumping program that is used to obtain account login and password information, normally in the form of a hash or a clear text password from an operating system.
THC Hydra
When you need to brute force crack a remote authentication service, Hydra is often the tool of choice. It can perform rapid dictionary attacks against more than 50 protocols, including telnet, ftp, http, https, smb, several databases, and much more.
WAR File
In software engineering, a WAR file (Web Application Resource[1] or Web application Archive[2]) is a file used to distribute a collection of JAR-files, JavaServer Pages, Java Servlets, Java classes, XML files, tag libraries, static web pages (HTML and related files) and other resources that together constitute a web application.
The Attack
The attacker finds an exposed Apache Tomcat Server for the organisation. This can be achieved in many ways, such as a simple Google search to show default configured Apache Servers:
The attacker browses to the associated Apache Tomcat server and see’s it is running up to date software and appears to be mainly left at default configuration:
The attacker attempts to access the Manager App, the manager app requires a username and password and therefore the attacker cannot login to make changes. Typically, these servers are setup with weak credentials:
Based off of this assumption, the attacker uses an application called THC Hydra to brute force the Tomcat Manager App using a list of passwords. After a short while, Hydra returns a successful set of credentials:
The attacker can now login to the Manager App using the brute forced credentials:
From here, the attacker can upload a WAR (Web application ARchive) file which contains their Web Shells:
The WAR file is nothing more than just a ZIP file with the JSP Web Shells inside. In this case, three Web Shells were uploaded:
After the upload, it is possible to see the new application called, admin (which is based off the WAR file name, admin.war), has been created:
The attacker has now successfully uploaded three Web Shells on to the server and can begin to use them. One of the Web Shells named, resetpassword.jsp, requires authentication to help protect direct access by other individuals; this page could also be adapted to confuse analysts when visited:
The attacker enters the password, and can begin browsing the web servers file system and executing commands, typical commands such as, whoami, are often used by attackers:
The attacker may also choose to see what processes are running to see if there are any applications that could hinder their progression by running, tasklist:
From the previous command, the attacker notices a lack of Anti-Virus so decides to upload Mimikatz via the WebShell:
The ZIP file has now been uploaded. This Web Shell also has an UnPack feature to decompress the ZIP file:
Now the ZIP file is decompressed:
The attacker can now use the Shell OnLine functionality within this Web Shell which emulates CMD in order to navigate to the Mimi directory and see their uploaded tools:
The attacker can then execute Mimikatz to dump all passwords in memory:
The attacker now has credentials from the Web Server:
The attacker could then use these credentials to laterally move onto other machines.
The attacker also dropped two other Web Shells, potentially as backups in case some get flagged. Let’s access those to see what they look like. This Web Shell is the JSP file called, error2.jsp, it has similar characteristics to the resetpassword.jsp Web Shell:
We can browse the file system and execute commands:
The final Web Shell uploaded, login.jsp, exhibits odd behavior when accessed:
It appears to perform a redirect to a default Tomcat Page named, examples, this appears to be a trick to confuse anyone who potentially browses to that JSP page. Examining the code for this Web Shell, it is possible to see it performs a redirect if the correct password is not supplied:
<SNIP>
Passing the password to this Web Shell as a parameter, which is defined at the top of this Web Shell’s code, we get the default response from the Web Shell:
Further analyzing the code, you can understand further parameters to pass in order to make the Web Shell perform certain actions, such as a directory listing:
This Web Shell is known as Cknife, and interacting it in this way is not efficient or easy, so Cknife comes with a Java based client in order to control the Web Shell. We can launch this using the command shown below:
The client is then displayed which would typically be used:
Note: This web shell is listed in this blog post as it is something the RSA Incident Response team consistently sees in some of the more advanced attacks.
The Analysis
Understanding the attack is important, and hence why it comes prior to the analysis section. Understanding how an attacker may operate, and the steps they may take to compromise a Web Server, will significantly increase your ability to detect these types of threats, as well as better understand the viewpoint of the analysis while triage is performed.
RSA NetWitness Packets
While perusing the network traffic, a large number of 401 authentication errors towards one the Web Servers was observed; there is also a large variety of what look like randomly generated passwords:
Focusing on the 401 errors, and browsing other metadata available, we can see the authentication errors are toward the Manager App of Tomcat over port 8080, also take note of the Client Application being used, this is the default from THC Hydra and has not been altered:
Removing the 401 errors, and opening the Filename and Directory meta keys, we can see the Web Shells that were being accessed and the tools that were uploaded:
NOTE: In an actual environment, a large number of directories and filenames would exist, it is up to the analyst to search for the filenames of interest that sound out of the norm or are in suspicious directories, are newly being accessed, and not accessed as frequently as other pages on the web server. For a more in-depth explanation to hunting using NetWitness Packets, take a look at the hunting guide available here: https://community.rsa.com/docs/DOC-79618
The analyst could also use other meta keys to look for suspicious/odd behavior. Inbound HTTP traffic with windows cli admin commands would be worth investigating, as well as sessions with only POST’s for POST based Web Shell’s, http post no get or http post no get no referer, for a couple of examples:
Investigating the sessions with windows cli admin commands yields the following two sessions, you’ll notice one of the sessions is one of the Web Shells, resetpassword.jsp:
Double clicking on the session will reconstruct the packets and display the session in Best Reconstruction view, in this case, web. Here we can see the Web Shell as the browser would have rendered it, this instantly should stand out as something suspicious:
This HTTP session also contains the error2.jsp Web Shell, from the RSA NetWitness rendering, it is possible to see the returned results that the attacker saw. Again, this should stand out as suspicious:
Coming back to the investigate view, and this time drilling into the sessions for post no get no referer, we can see one of the other Web Shells, login.jsp:
Double clicking on one of these sessions shows the results from the Cknife Web Shell, login.jsp:
As this was not a nicely formatted web based Web Shell, the output is not as attractive, but this still stands out as suspicious traffic: why would a JSP page on a Web Server return a tasklist?
Sometimes changing the view can also help to see additional data. Changing the reconstruction to text view shows the HTTP POST sent, this is where you can see the tasklist being executed and the associated response:
Further perusing the network traffic, it is also possible to see that Mimikatz was executed:
This is an example of what the traffic may look like in RSA NetWitness Packets. The analyst would only need to pick up on one of these sessions to know their organization has been compromised. Pro-active threat hunting and looking for anomalies in traffic toward your web servers will significantly reduce the attacker dwell time.
The activity observed was only possible due to the communication happening over HTTP. If this had been SSL, the detection via packets would be much more difficult. This is why introducing SSL Decryption/Interception/Offloading is highly recommended. SSL inspection devices are nothing more than a well-designed man-in-the-middle attack that breaks the encryption into two separate encrypted streams. Therefore, they still provide an adequate level of protection to end-users while allowing security analysts and devices to properly monitor and alert when malicious or unwanted activity takes place, such as the web shells shown here. In summary, if you are responsible for protecting your organization’s assets, you should definitely consider the pros and cons of using this technology.
RSA NetWitness Endpoint
On a daily basis, analysts should be perusing the IIOC’s within NWE, paying particular attention to the Level 1 IIOC’s. Upon logging into NWE, we can see an interesting Level 1 IIOC has hit, HTTP Daemon Runs Command Shell. This IIOC is looking for a HTTP Daemon, such as Tomcat, spawning cmd.exe:
If we double click on the machine name in the Machines window, we can then navigate to the tracking data for this machine to see what actually happened. Here we can see that Tomcat9.exe is spawning cmd.exe and running commands such as whoami and tasklist, this is not normal functionality and should raise concern for the analyst. We can also see the Mimikatz execution and the associated command executed for that:
Another IIOC that would have led us to this behavior is, HTTP Daemon Creates Executable:
Again, coming back into the tracking data, we can see the Tomcat9.exe web daemon writing files. This would be note-worthy and something that should be investigated further, as web daemons can perform this activity normally. In this instance, the presence of Mimikatz is enough for us to determine this would be malicious activity:
The analyst also has the capability to request files from the endpoint currently under analysis by right-clicking on the machine name, selecting Forensics and then Request File(s):
The analyst can specify the files they want to collect for their analysis (NOTE: wildcards can be used for the filenames but not directories). In this case, the analyst wants to look into the Tomcat Access files, and requests that five of them be returned:
Once the files have been downloaded, the analyst can save them locally by navigating to the Download tab, right-clicking the files of interest and selecting Save Local Copy:
Perusing the access files, the analyst can also see a large number of 401 authentication errors to the Tomcat Web Server which would have been from the THC Hydra brute force:
And also evidence of the Web Shells themselves. Some of the commands the attacker executed can be seen in the GET requests, the data in the body of the POST's however, does not show in the log file, showing why it is important to have both Packets and Endpoint visibility to understand the interaction with the Web Shell:
Conclusion
Understanding the potential areas of compromise within your organisation vastly increases your chances of early detection. This post was designed to show one of those potential areas of importance for attackers, and how they may go about a compromise, while also showing how that attack may look as captured by the RSA NetWitness Platform. It is also important to understand the benefits behind proactively monitoring the RSA NetWitness products for malicious activity, simply awaiting for an alert is not enough to capture attacks in their early stages.
It is also important to for defenders to understand how these types of attacks look within their own environment, allowing them to better understand, and subsequently protect against them.
This is something our RSA Incident Response practice does on a daily basis. If your organization needs help or your interested to learn more, please contact your account manager.
As always, happy hunting!
