SDK Content Command

One of the powerful commands in NwConsole is sdk content. It contains numerous options to do just about anything, at least as far as extracting content from the NetWitness Core stack. You can use it to create PCAP files, log files, or extract files out of network sessions (for example, to grab all of the pictures from email sessions). It can append files, have a maximum size assigned before creating a new file, and automatically clean up files when the directory grows too large. It can run queries in the background to find new sessions. It breaks queries into manageable groups and performs those operations automatically. When the group is exhausted, it does a requery to get a new set of data for further operations. The list of options for the SDK content command is very extensive.

Because the command has so many options, this document provides examples of commands for different use cases.

Before you can run sdk content, there are a few commands (like logging into a service) that you need to run first. Here are some examples:

  • First connect to a service:
    sdk open nw://admin:netwitness@10.10.25.50:50005
  • If you need to connect over SSL, use the nws protocol:
    sdk open nws://admin:netwitness@10.10.25.50:56005
  • Keep in mind that you are passing a URL and must URL encode it properly. If the password is p@ssword, the URL looks like this:
    sdk open nw://admin:p%40ssword@10.10.25.50:50005
    This also applies to username.
  • Once you log in, you can set an output directory for the commands: sdk output <some pathname>
  • For command line help, type: sdk content

Before you try the example commands, it is important to understand the sessions parameter. This parameter is very important and controls how much or how little data you want to grab (the where clause is also important). The sessions parameter is either a single session ID or a range of session IDs. All NetWitness Core services work with session IDs, which start at 1 and increase by 1 for every new session added to the service (network or log session). Session IDs are 64-bit integers, so they can get quite large.

For example, to keep it simple, assume we have a Log Decoder that has ingested and parsed 1000 logs. On the service, you now have 1000 sessions with session IDs from 1 to 1000 (session ID 0 is never valid). If you want to operate over all 1000 sessions, you pass sessions=1-1000. If you only want to operate over the last 100 sessions, you pass sessions=901-1000. Once the command finishes processing session 1000, it exits back to the console prompt.

Many times, however, we do not care about specific session ranges. We just want to run a query over all of them and process the sessions that match a query. Here are some shortcuts that simplify this:

  • The letter l (lowercase L) means lower bound or the lowest session ID.
  • The letter u means the highest session ID. In fact, it actually means the highest session ID for future sessions as well. In other words, if you pass sessions=l-u, this special range means operate over all the current sessions in the system, but also do not quit processing, and as new sessions enter the system, process those, too. The command pauses and waits for new sessions once it reaches the last session on the service. To summarize, the command never exits and goes into continuous processing mode. It runs for days, months, or years, unless it is killed.
  • If you do not want the command to run forever, you can pass now for the upper limit. This determines the last session ID on the service at the time the command starts and processes all sessions until it reaches that session ID. Once it reaches that session ID, the command exits, regardless of how many sessions may have been added to the service since the command started. So, for the example Log Decoder, sessions=200-now starts processing at session 200 and goes all the way to session 1000 and quits. Even if another 1000 logs were added to the Log Decoder after the command started, it still exits after processing session 1000.
  • The parameter sessions=now-u means start at the very last session and continue processing all new sessions that come in. It does not process any existing sessions (except the last one), only new sessions.
  • If you want to run this continually and have it remember the last session it processed between invocations, use the sessionPersist={pathname} parameter. The pathname should be a valid filename where the command will write the last successful session id. Upon restart, it will read the last session id and pick up where it left off.

For example commands and what they do, type man sdk content examples or see SDK Content Command Examples.