Preface
In order to prevent confusion, I wanted to add a little snippet before we jump into the analysis. The blog post
first goes over how the server became infected with Metasploit, it was using a remote execution CVE
against an Apache Tomcat Web Server, the details of which can be found here, https://nvd.nist.gov/vuln/detail/
CVE-2019-0232. Further into the blog post, details of Metasploit can be seen.
This CVE requires that the CGI Servlet in Apache Tomcat is enabled. This is not an abnormal servlet to be
enabled and merely requires the Administrator to uncomment a few lines in the Tomcat web.xml. This is a
normal administrative action to have taken on the Web Server:
Now, if the administrator has a .bat, or .cmd file in the cgi-bin directory on the Apache Tomcat Server. The
attacker can remotely execute commands as Apache will call cmd.exe to execute the .bat or .cmd file and
incorrectly handle the parameters passed; this file can contain anything, as long as it executes. So here as an
example, we place a simple .bat file in the cgi-bin directory:
From a browser, the attacker can call the .bat file and pass a command to execute due to the way the CGI
Servlet handles this request and passes the arguments:
From here, the attacker can create a payload using msfvenom and instruct the web server to download the Metasploit payload they had created:
The Detection in NetWitness Packets
RCE Exploit
NetWitness Packets does a fantastic job pulling apart the behaviour of network traffic. This allows analysts to
detect attacks even with no prior knowledge of them. A fantastic meta value for analysts to look at is windows
cli admin commands, this metadata is created when cli commands are detected; grouping this metadata with inbound
traffic to your web servers is a great pivot point to start looking for malicious traffic:
NOTE: Taking advantage of the traffic_flow_options.lua parser would be highly beneficial for your SOC. This parser allows you to define your subnets and tag them with friendly names. Editing this to contain your web servers address space for example, would be a great idea.
Taking the above note into account, your analysts could then construct a query like the following:
(analysis.service = 'windows cli admin commands') && (direction = 'inbound') && (netname.dst = 'webservers')
Filtering on this metadata reduces the traffic quite significantly. From here, we can open up other meta
keys to get a better understanding of what traffic is related to these windows cli commands. From the below
screenshot, we can see that this is HTTP traffic, with a GET request to a hello.bat file in the /cgi-bin/ directory,
there are also some suspicious looking queries associated with it that appear to reference command line
arguments:
At this point, we decide to reconstruct the raw sessions themselves as we have some suspicions surrounding
this traffic to see exactly what these HTTP sessions are. Upon doing so, we can see a GET request with the
dir command, and we can also see the dir output in the response - this will be what the windows cli admin
commands metadata was picking up on:
This traffic instantly stands out as something of interest and as being something that requires further
investigation. In order to get a holistic view of all data toward this server, we need to reconstruct our query, as
the windows cli admin commands metadata would have only picked up on the sessions where it saw CLI
commands, we are, however, interested in seeing it all. So we look at the metadata available for this session
and build a new query. This now allows us to see other interesting metadata and get a better idea of what the
attacker was doing. Looking at the Query meta key, we can see all of the attackers commands:
Navigating to the Event Analysis view, we can see the commands in the order they took place and reconstruct
what the attacker was doing. From here we can see a sequence of events whereby the attacker makes a
directory, C:\temp, downloads an executable called 2.exe to said directory, and subsequently executes it:
MSF File and Traffic
As we can see the attackers commands, we can also see the download for an executable they performed, a.exe. This means we can run a query and extract that file from the packet data as well. We run a simple query looking for a.exe
and we find our session. Also, take note of the user agent, why is certutil being used to download a.exe? This is also a great indicator of something suspicious:
We can also choose to switch to the File Analysis view and download our file(s). This would allow us to perform additional analysis on the file(s) in question:
Merely running a strings on one of these files yields a result of a domain this executable may potentially connect to:
As we also have another hostname to add to our analysis, we can now perform a query on just this hostname
to see if there is any other interesting metadata associated with it. Opening the session analysis meta key, we can see a myriad of interesting pivot points. We can group these pivot points together, or make combinations of them to whittle down the traffic to something more manageable:
NOTE: See the RSA IR Hunting guide for more details on these metadata values: https://community.rsa.com/docs/DOC-62341
Once we have pivoted down using some of the metadata above, we start to get down to a more manageable amount of sessions - continuing looking at the service analysis meta key we also observe some additional pieces of metadata of interest we can use to start reconstructing the sessions to get a better understanding of what this traffic is:
- long connection
- http no referer
- http six or less headers
- http post missing content-type
- http no user-agent
- watchlist file fingerprint
Opening these sessions up in the Event Analysis view, we can see an HTTP POST with binary data, and a 200 OK from the supposed Apache Server, we can also see the directory is the same as we saw from our strings analysis:
Continuing to browse through these sessions, yields more of the same:
Navigating back to the investigate view, it is also possible to see that the directory is always the same and the one we saw in our strings analysis:
NOTE: During the analysis, no beaconing pattern was observed, this can make the C2 harder to detect and requires continued threat hunting from your analysts to understand your environment and pick up on these types of anomalies.
Web Shell
Now we know that the Apache Tomcat Web Server is infected, we can look at all other traffic
associated with the Web Server and continue to monitor to see if anything else takes place, attackers like to keep
multiple entry points if possible. Focusing on our Web Server, we can also see a JSP page being accessed
that sounds odd, error2.jsp, and observe some additional queries:
Pivoting into the Event Analysis view and reconstructing the sessions, we can see a tasklist command being
executed:
And the subsequent response of the tasklist output. This is a Web Shell that has been placed on the server and
the attacker is also using to execute commands:
NOTE: For more information on Web Shells, see the following series: https://community.rsa.com/community/products/netwitness/blog/2019/02/12/web-shells-and-netwitness
It is important to note that just because you have identified one method of remote access, it does not mean that
is the only one, it is important to ascertain whether or not other access methods were made available by the
attacker.
The Detection in NetWitness Endpoint
As I preach in every blog post, the analyst should always log in every morning and check the following
three meta keys as a priority, IOC (Indicators of Compromise), BOC (Behaviours of Compromise), and EOC
(Enablers of Compromise). Looking at these keys, a myriad of pieces of metadata stand out as great places to
start the investigation, but let's place a focus on these three for now:
Let's take the downloads binary using certutil to start, and pivot into the Event Analysis view. Here we
can see the certutil binary being used to download a variety of the executable we saw in the packet data:
Looking into one of the other behaviours of compromise, http daemon runs command shell, we can also
see evidence of the bat file being requested and the associated commands, as well as the use of the Web
Shell, error2.jsp. It is also important to note that there is a request for the hello.bat prior to the remote code
execution vulnerability being exploited, this would be seen as legitimate traffic given that the server is working
as designed for the CGI-BIN scripts. It is down to the analyst to review the traffic and decipher whether or not
something malicious is happening, or whether this is by design of the server:
NOTE: Due to the nature of how the Tomcat server handles the vulnerable cgi-bin application and "legitimate" JSP files, you can see hello.bat as part of the tracking event as it's an argument passed to cmd.exe. However, with the error2.jsp, it is executed inside the Tomcat process, and only when the web shell spawns a new command shell to execute certain commands will you see cmd.exe being executed, and not every time error2.jsp is used. Having said that, the advantage for the defender is that even if not all of it is tracked, or leaves a visible footprint, at some point something will, this will/ could be the starting thread needed to detect the intrusion.
Coming back to the Investigate view we can see another interesting piece of metadata that would be of interest, creates remote service - let's pivot on this and see what took place:
Here we can see that cmd was used to create a service on our Web Server that would run a malicious binary dropped by the attacker in the c:\temp directory:
It is important to remember that as a defender, you only need to pick up on one of these artifacts leftover from
the attacker in order to start unraveling their activity.
Conclusion
With today's ever-changing landscape, it is becoming increasingly inefficient to create signatures for known
vulnerabilities and attacks. It is therefore far more important to pick up on behaviours of traffic that stand out as
abnormal, than generating signatures. As shown in this blog post, a fairly recent remote code execution CVE
was exploited, https://nvd.nist.gov/vuln/detail/CVE-2019-0232 - no signatures were required to pick up on this
as NetWitness pulls apart the behaviours, we just had to follow the path. Similarly, with Metasploit it is also very difficult to generate effective long life signatures that could detect this C2; performing
threat hunting through the data based on a foundation of analysing behaviours, will ensure that all knowns and
unknowns are effectively analysed.
It is also important to note that the packet traffic would typically be encrypted but we kept it in the clear for the purposes of this post, with that being said, the RCE exploit and Web Shell is easily detectable when NetWitness Endpoint tracking data is being ingested, and this allows the defender to have the necessary visibility if SSL decryption is not in place.
