Investigate using Archiver Issue

[yazantaleb01]

Hello,

Recently, I've enabled the meta key 'action' on the archiver so that I can use it to filter the results in the Investigate window, I've done that by modifying the file 'index-archiver-custom.xml', adding a new key for it.

I've applied the new changes and restarted the aggregation service which went well.

Now, whenever I try to investigate using the Archiver, I got empty results as the following screenshot:

Thanks in Advance.

[JohnKisner]

@yazantaleb01 Please do not Investigate against the Archiver. This is not supported. You should only be using the Reporting Engine to do reports against the archivers. The system allows you to Investigate against it but your performance may vary dramatically and the interface may time out. This is why the Archivers are only for reporting purposes.

The following article shows how to fully add new meta to the archiver. The article is dated to 10.6.x but the steps are the same in the newer versions of NetWitness. https://community.netwitness.com/t5/netwitness-knowledge-base/how-to-add-additional-meta-keys-to-the-rsa-netwitness-archiver/ta-p/677579

Not only do you have to update the index-archiver-custom.xml, which is used to create indexing, but you also have to tell the archiver to start retrieving the meta from the log decoder.

1. Go into the Archiver's Config -> General page.

2. Toggle the Archiver's service that is connected to the log decoder you want to retrieve the new meta from so that is shows as offline in the Status column.

3. Click on the Edit Aggregate Service button, it is the square box with the red pencil in it.

4. In the box that pops up go to the Meta Include tab and select the meta you want to retrieve. 

Do not select all meta, only select the meta that you want to index, otherwise you can drastically reduce the retention time of your archiver.

5. Once all the meta that you want to collect is selected, click Save.

6. Toggle the service back on so it is consuming from the log decoder again.

If you have more than one log decoder on an archiver that has the meta that you want, you'll need to do the same steps to all the connected log decoders not just the first one connected. You can see if the meta you want to collect is now saved by clicking on the little black circle with the i in it under the Meta Include column for the connected log decoder. The popup shows all the meta that the archiver is pulling from that log decoder.

I hope this helps. Please review the linked article for more information.

[yazantaleb01]

Thanks for the valuable information.

My point is that I want to investigate and apply some filters on old events that are only available using the Archiver, for example, 3 months back which is not available on the Concentrator. Is there any other way to achieve this?

Regards.

[JohnKisner]

@yazantaleb01  Yes you should create the query as a rule in the Reporting Engine. Then run that rule as part of a report against the archiver. It will return the results you are looking for. Just remember that whatever meta that is associated with the logs you are searching for you need to put in the Select field of the reporting engine rule.

If you can provide some more detail as to exactly what you are attempting to find I might be able to provide an example of what I mean. Just remember any query you can do in Investigation you can do with a reporting engine rule.

[yazantaleb01]

Hi @JohnKisner ,

I tried to generate some reports using the reporting engine, but unfortunately I got the same issue, there is no available data:

So, I believe that the problem is basically with the Archiver service.

Here are the actions I did so far:

I tried to force reindexing on the archiver (reset index=1)

I tried to reinstall the archiver service on the host

I tried to restore the default meta keys

but still same issue. I noticed something else, whenever I try to modify the file 'index-archiver-custom.xml' and click push button I get this error message:

Is this related to the basic problem we have? Is there any way to restore a clean version of this file? any way to regenerate this file? what happens if I deleted it?

Thanks in advance.

[JohnKisner]

Are you indexing meta keys that are not a part of the default 42 keys? If not then the index-archiver-custom.xml file doesn't contain anything. It is just a file with comments in it.

How did you try to modify the custom index xml?

What is the rule that you are using in the Reporting Engine report? Can you provide the exact syntax or provide a screenshot of the full rule page?

When you forced the reindexing did you wait until it was completed before doing any further items? Reindexing can take some time depending on the amount of data that is in the archiver.

Can you provide the output for df -h command?

If you feel comfortable, can you provide the /var/log/messages file?

Please do not attempt to reinstall anything at this point as it could make things worse depending on what the exact issue is.

[yazantaleb01]

Hi @JohnKisner 

• The meta key that I enabled are "Action Event, Event Description", I will share the custom index file directly with you.
• I updated the custom index file using the UI (Services -> Archiver -> config -> Files)
• Whatever rule I use, there are no results. I didn't apply any filter in the rule.

• I waited for the reindex process to completely finish, based on the status of the aggregation status (consuming), it took multiple hours to finish.
• Here is the output of 'df -h' command

• I will share the messages file privately with you via direct message.

Thanks.

[JohnKisner]

Well the first issue with the Reporting rule is you have no Where clause. Without that there is nothing telling the reporting engine what you are looking for. Just setting the select section will not produce any results. What is it that you are trying to find, as that is what is needed in the Where clause. The Select field is what is used to tell the system what meta keys you want to display based on Where clause query. Think of the Where clause like the query you use in Investigation to find what you are looking for.

The Where clause could be something as simple as 'action exists' or 'action = login', etc. I'll take a look at the items that you sent me as well just to make sure there is not anything else that may be incorrect.

[JohnKisner]

@yazantaleb01 Unfortunately the private message you sent will not allow me to view either the video or the directly link that was provided. 

[yazantaleb01]

I tried multiple where conditions but with no result, simply used 'did exist' in the where clause which refers to the decoder ID and no result as well.

I applied that same rules on different data source such as the concentrator and it worked fine.