Raw logs and Meta from Archiver to Splunk

[VishamRawat]

We've got a requirement to move all our raw logs and meta stored on the Archiver to the Splunk platform.

Now, I see there's a document on the Community that speaks of RSA NetWitness and Splunk.

I’ve gone through the document. I find procedures to pivot investigations from RSA NetWitness to Splunk and vice versa. I also see procedures to forward NetWitness audit/security logs, ESA and RE alerts to Splunk.

However, what I’m looking for is to migrate all stored event log data [on my Archivers, whatever the duration] to Splunk. I don’t see a procedure for this.

Any insights on how this can be achieved?

[DaveGlover]

Visham

You are going to need to run a rest query to extract the data out of the archiver. That is the only way to bulk extract the data from the Archiver

Dave

[VishamRawat]

Hi Dave,

Thanks for the response. I accessed the REST API for the Archiver via port 50106. I do see a Logs folder there, but it contains only System logs [authentication]. Where do I find the device logs?

[AaronMartin2]

You need to use the REST API query as if you were trying to run an investigation using that. You would need to make SDK-packet calls to get the raw logs for usage and have them written to a file. I suggest you look at the REST API docs for that.

I will provide another alternative but I must admit I have not tried it outside of this example. You can use NwConsole on the Archiver to use the makepcap command to convert packetdb files to log files, in this case:

For detailed help, type "help <command>"
> makepcap
Usage: makepcap {source=<pathname>} [dest=<pathname>] [filenum=#[-#]]
                [packetid=#[-#]] [time1=<time>] [time2=<time>] [ip=<IP
                Address>] [single=<0,1>] [gzip=<0,1>]
                [fileType=<pcap,pcapng,log,json>] [delimiter=<string>]
                [delete=<0,1>]
Convert packet database files to pcap or log files

    source    - Required, the directory where the packet db files reside
    dest      - The destination directory for pcap files, uses source dir by
                default
    filenum   - The packet db file numbers to operate on or range of file
                numbers, use 999999999 for no limit on upper range
    packetid  - Only extract the packet IDs within the specified range
    time1     - Start time (UTC) for all packets to be extracted from packet db
                (e.g., time1="2009-Dec-17 08:00:00")
    time2     - Stop time (UTC) for all packets to be extracted from packet db
                (e.g., time2="2009-Dec-17 08:05:00")
    ip        - Limit to a single IP address, will match both source and
                destination
    single    - Create a single output file if 1, 0 is default
    gzip      - Compress output files if 1, 0 is default
    fileType  - pcap, pcapng, log or json, default is pcap. If json, it will
                write out single json objects, one per line, with packet
                metadata in them.
    delimiter - Log output line ending, default is \n (e.g., for windows you
                can use \r\n)
    delete    - If true, deletes the db file after processing, default is false
> makepcap source=/var/netwitness/archiver/database0/default/packetdb dest=/tmp/archiverpcap filenum=1 fileType=log
7 files found
packet database file: packet-000000001.nwpdb, starting packet ID is 1
 - converting to file: packet-000000001.log
 - creating new file
 - wrote 485979 packets (1073741824%)   timestamp: 08/02/2019 06:36:53 PM
 - finished writing 485979 packets in 10 secs
 - 6420 packets written out of time sequence

>

Just be very careful when working through this. You don't want to fill up the file system while working from this. This is why I use filenum=1 so it only goes through the first packetdb file in my case.

You can do something similar with makemeta to get the meta but I do not know how Splunk will ingest either section of data.

I hope this helps.

[VishamRawat]

Hi Aaron,

For some reason, I've been able to access the REST API UI of the Archiver, and I just can't find the /sdk node. It doesn't exist. I can see a list of other nodes like /logs, /appliance, /services, etc. but not /sdk. I believe the raw logs and meta are stored within the /sdk node.

What can be the reason for this? And how do I run queries against this node, when it's not there?

[AaronMartin2]

I believe 1 of 2 things is going on here. Either you are logging in with a user who lacks the permissions to perform those operations ( I don't think is true yet) or you are connecting to the wrong REST interface. Port 50106 is the default rest port for nwappliance while Archiver is setup on 50108. Could this be your problem?

[JoshRandall]

will all the data in the archiver's collections be available through the top-level SDK node? or will it need to be accessed through each collection's SDK node?

[AaronMartin2]

Good point to make. Top level /sdk will run a query against all collections while /archiver/collections/<collectionName>/sdk will run queries only for that collection, meaning, only the data within that collection. This is a good way to narrow your search.

[VishamRawat]

Hi Aaron,

Do I need to enable access to port 50108 on the Archiver, in some configuration setting on the UI/appliance?

Because right now, all I'm able to access is port 50106, and not 50108.

[Nauman]

Hi Visham Rawat, were you able to complete this task? Can you share the experience for the folks in the same boat. Thank you