Bulk Data Export From RSA Netwitness Archiver

[DevarajMohan1]

Hi Team,

Currently we using RSA Netwitness 11.3.1.1 in our organization. So we have archiver which is deployed for log retention. At present we having 5 months of log data stored in archiver. We have the below requirement.

1) Management is asking 3 months of log data from archiver in human readable format lets say in .log or .csv format. Kindly suggest us on this. We haven't configured any aggregation polices on archiver for data collection. By default all the log data s are get aggregated.

2) If log data export is not possible what is the mechanism to read the archiver data ? Is there any component we need to deploy for reading archiver data ?

Looking forward for response.

Regards,

Devaraj

Counsultant (Cognitive SOC)

Inspirisys Solutions Limited, India.

[sravan.koneti]

Devaraj Mohan‌,

Try https://community.rsa.com/docs/DOC-45813 by replacing the archiver details and port 50008

[MarcoDemonte1]

Hello Devaraj,

probably the best way is to use Reports.

If you extract raw data from the archiver, you will have the logs as they are produced by the log sources. I don't think that the managment is interested in those data.

Quite the opposite with reports where (for example) you can choose the data sources (device.type) and include some useful meta.key (user.dst or ip.src) as long as the log itself (msg)

The output could be pdf or csv. It depends on the ampount of data but you can use Excel for some csv.

If you really really really need to extract all the raw logs, I had a similar need in the past and I successfully used the sdk command from the archiver nwconsole. It is the third method mentioned by Sravan and that guide explains it very well.

I hope I've been of some help.

kind regards

Marco

[DevarajMohan1]

Sravan Kumar Koneti‌ As per the document you have shared is fine and we tested step 1 procedure. When we tried to export logs in CSV format it is not exporting instead i am getting below out put in browser. Kindly help us to download logs from archiver in CSV format.

Action:-

Output:-

[sravan.koneti]

Devaraj Mohan‌,

The tab contents are acutal csv format logs. you can right click and save as .csv file. The columns for csv are timestamp,source,forwarder,lccid,log. The log coulmn is actual raw log.

Also, when you use GUI or REST 1 GB is limit. Going for 3rd method gives unlimited logs.

[AgustinGras]

Hello @sravan.koneti , I need to get raw logs from the archiver and I found the sdk method that you shared the best option for me. The issue that I am having is that when I do the sdk query I am having this message: "(W) 2021-Aug-03 14:26:15 [ClientChannel] Channel (407515 -> 118) has too many queued messages and is unresponsive. Closing channel.". After that the logs are written but not all of them. I am working on a 11.2 Nw environment

Do you know why this could be happening?

Regards

[sravan.koneti]

Hi @AgustinGras ,

It would be for query response messages (results on its channel) that would exceed more than 1000 and when channel becomes unresponsive for the case of streaming query, then Broker would close the channel. You would see similar log message on Brokers   [ClientChannel] [warning] Channel (..) has too many queued messages and is unresponsive. Closing channel. 

I also noticed that you are running this queries on Broker. Can you please try running on Archiver directly.

[AgustinGras]

Hi @sravan.koneti , thank you for your help.

Yes, I tried on the archiver too but I had the same issue, only with the difference that in the broker more logs where written in the output file.

Another issue that I have is the "28800000 ms timeout reached waiting for server response" message

This happens on broker and archiver. 

Regards