Extracting Files using NwConsole

[RSAAdmin]

A little known feature in NextGen, that has been around since NextGen v9.0, is a console command that will run unattended and extract files from packet sessions based on criteria you enter.

The enhancements to NwConsole are in the sdk content calls. So run NwConsole, and login to a collection:

sdk open nw://username:password@concentrator:50005

sdk info - will give you some stats about the system including the session IDs

Set your output directory:

sdk output somedirectory

Then issue your content call. Type "sdk content" to get the options:

sdk content [options]

options:

session=[#]|[#-#] The session or session range (optional) to extract content from

where={where clause} The where clause (optional), used to determine sessions to extract

render=# The render type (defined in NwSDK.h) for content or render name

flags=# The content flags, zero is default

size=# The maximum session size to retrieve, unlimited is default

dir={pathname} Directory path where content files will be placed

maxDirSize=# Max directory size in MBs, default is unlimited

includeFileTypes=.ext1;.ext2 Semicolon separated list of file extensions that will be extracted

excludeFileTypes=.ext1;.ext2 Semicolon separated list of file extensions that will be excluded

renameFileTypes=.ext1|.ext2 Semicolon and pipe separated list of file extensions that will be renamed

render can be a number (defined in NwSDK.h) or one of the following render types:

text, hex, packets, web, mail, raw, rtp, voip, meta, im or files.

renameFileTypes is used to rename certain files from one or more extensions to another. For example:

renameFileTypes=.download|.octet-stream|.program|.exe;.jpeg|.jpg

For the above example, all files ending in .download, .stream or .program will be renamed to .exe

All files ending in .jpeg will be renamed to .jpg

WARNING: Setting maxDirSize will scan the output directory every 20 minutes and will

indiscriminately delete the oldest files that exceed the threshold. Please do not use a directory

with existing files that should not be deleted!

To run continuously, you must provide a where clause and do not provide a session range.

Or you can provide a lower bound session id but leave the upper session id unbounded like:

sessions=1000-u Start at session 1000 and continue nonstop

sessions=now-u Means extract only new sessions as they come in

sessions=2000-3000 Extract sessions between 2000 and 3000 then quit

So a command such as this...

sdk content session=now-u where="extension=exe,dll" render=files includeFileTypes=.exe;.dll; maxDirSize=1000

...will extract all exe and dll files from any session where there is a registered extension of exe or dll. Obviously - we are looking for executables that are not always properly named. So you could use a combination of the filetypes.parser file, located in the content pack. Then issue the following:

sdk content session=now-u where="alert=file_signature_windows_executable" render=files includeFileTypes=.exe;.dll; maxDirSize=1000

Some other examples:

PDFs

sdk content session=now-u where="extension=pdf" render=files includeFileTypes=.pdf maxDirSize=1000

MP3s

sdk content session=now-u where="extension=mp3" render=files includeFileTypes=.mp3; maxDirSize=1000 renameFileTypes=.octet-stream|.mp3

Images

sdk content session=now-u where="extension=jpg,jpeg,png,bmp,gif" render=files includeFileTypes=.jpg;.jpeg;.png;.bmp;.gif maxDirSize=1000

Documents

sdk content session=now-u where="extension exists" render=files includeFileTypes=.doc;.docx;.xls;.xlsx;.ppt;.pptx; .pdf;.zip maxDirSize=1000

You can throw all the commands in a text file, and just launch NwConsole with the -f file option to automate.

Now - for the obligatory warnings:

First - this uses the EXISTING content reconstruction. If the file fails to render in Investigator, it will probably not be extracted correctly. What is happening, is that the SDK uses the index to find matching sessions, and then asks the decoder to reconstruct that session, along with ALL files. It then sends that reconstructed session back to the client, where only the requested file types are extracted and saved. This has the unintended effect of caching that session on the decoder. The same process happens behind the scenes in investigator. This just does it a lot quicker, and at higher volumes.

If you have any questions, please respond to this post.

Enjoy!

Scott

[Anonymous]

Thanks for the great information Scott!

I'm looking for the same level of detail for log data and specifically fro NWFL.

Any help would be greatly appreciated.

[RSAAdmin]

With the newer versions, you can specify render types like pcap, nwd or logs.  So for NWFL, if you want to out a plain text file of matching logs, use the render=logs parameter.

[CHADHEILIG]

Getting 'collection not open' when running sdk:

[RSAAdmin]

That means you didn't open correctly with "sdk open".  It would be helpful if you would show the whole console session.

[RSAAdmin]

While there could be a dozen reasons for a collection open failure, I'll mention for the sake of posterity and discussion that when passing creds via the sdk open URL ("sdk open nws://user:pass@url:port"), you may need to URL-encode the password.  If using complex or generated passwords, this is a good thing to keep in the back of your mind lest you drive yourself mad having confirmed for the umpteenth time that yes, the bloody password is correct...

[Anonymous]

Hi Scott,

Is there a way we can pull user list from SA using Nwconsole? Or to add new user in VLC using NwConsole?

thanks.

[RSAAdmin]

Here's a sample NwConsole session where I get a directory listing of the current accounts, add a new user, then delete the user I just added.

RSA Security Analytics Console 10.5.0.0.0

Copyright 2001-2014, RSA Security Inc.  All Rights Reserved.

> login

login <server:port[:ssl]> <username> <password>

> login 192.168.1.70:50002 admin

Password: **********

Successfully logged in as session 3819

[192.168.1.70:50002] /> cd /users/accounts

[192.168.1.70:50002] /users/accounts

[192.168.1.70:50002] /users/accounts> ls

admin

[192.168.1.70:50002] /users/accounts> ls depth=3

admin

config

auth.type (Authentication Type) = netwitness

description (Description) = Administrator account for this service

display.name (Display Name)

email (Email Address)

groups (Groups) = Administrators

password (Password) = ********

query.level (Query Level) = 3

query.prefix (Query Prefix)

session.threshold (Session Threshold) = 0

stats

last.login.failure.time (Last Login Failure Time)

last.login.time (Last Successful Login Date) = 2014-Nov-06 17:00:12

login.failures (Login Failures) = 0

[192.168.1.70:50002] /users/accounts> send /users help

[description: Folder containing all users of the system

security.roles: everyone,users.manage

message.list: The list of supported messages for this node

ls:  [depth:<uint32>] [options:<string>] [exclude:<string>]

mon:  [depth:<uint32>] [options:<uint32>]

info:

help:  [msg:<string>] [op:<string>] [format:<string>]

auths:

count:

delete:  name:<string>

unlock:  name:<string>

stopMon:

addOrMod:  name:<string> [password:<string>] [pwdIsHashed:<bool>] [groups:<string>] [authType:<string>] [queryLevel:<uin

t32>] [displayName:<string>] [email:<string>] [description:<string>] [queryPrefix:<string>] [sessionThreshold:<uint32>]

]

[192.168.1.70:50002] /users/accounts> send /users help msg=addOrMod

[addOrMod: Add a new user or update an existing user in the system

security.roles: users.manage

parameters:

   name - <string, {char:abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789@!#$%&'+-=?^_`{|}~.[]}> The usern

ame to add or update, must be alphanumeric or @!#$%&'+-=?^_`{|}~.[]

   password - <string, optional> The user's password

   pwdIsHashed - <bool, optional> Set to true if the password has already been hashed

   groups - <string, optional> The groups the user belongs to

   authType - <string, optional, {enum-one:netwitness|pam}> The authentication system to use, "netwitness" is the defaul

t

   queryLevel - <uint32, optional, {range:1 to 3}> The query priority level, maximum running times per level are defined

under /sdk/config

   displayName - <string, optional> Display name for this user account

   email - <string, optional> Email address for this account

   description - <string, optional> Description of this user account

   queryPrefix - <string, optional> Query filter applied to every query performed by this user account

   sessionThreshold - <uint32, optional, {range:0 to 4294967295}> Query optimization which will extrapolate the remainin

g session counts when they exceed this value

]

[192.168.1.70:50002] /users/accounts> send /users addOrMod name="TestUser" password="ThisIsAPassword" groups="Analysts" queryLevel=1 sessionThreshold=100000

The user TestUser was added or modified successfully

[192.168.1.70:50002] /users/accounts> ls

TestUser

admin

[192.168.1.70:50002] /users/accounts> ls depth=3

TestUser

config

auth.type (Authentication Type) = netwitness

description (Description)

display.name (Display Name)

email (Email Address)

groups (Groups) = Analysts

password (Password) = ********

query.level (Query Level) = 1

query.prefix (Query Prefix)

session.threshold (Session Threshold) = 100000

stats

last.login.failure.time (Last Login Failure Time)

last.login.time (Last Successful Login Date)

login.failures (Login Failures) = 0

admin

config

auth.type (Authentication Type) = netwitness

description (Description) = Administrator account for this service

display.name (Display Name)

email (Email Address)

groups (Groups) = Administrators

password (Password) = ********

query.level (Query Level) = 3

query.prefix (Query Prefix)

session.threshold (Session Threshold) = 0

stats

last.login.failure.time (Last Login Failure Time)

last.login.time (Last Successful Login Date) = 2014-Nov-06 17:00:12

login.failures (Login Failures) = 0

[192.168.1.70:50002] /users/accounts> send /users delete name="TestUser"

The user 'TestUser' was deleted successfully

[192.168.1.70:50002] /users/accounts> ls

admin

[Anonymous]

This is so good, Really helpful. Much appreciated. thanks. 

[Anonymous]

The json export works fine on the web interface http://<concentrator>:50105/sdk/packets.

However, the NwConsole export fails with render=json or render=application/json...

Any tips to export logs in json with NwConsole ?