Creating a "RAW LOG" Meta Key to Speed up Searching in Investigate?

[ChrisIchelson]

Has anyone tried to create a custom META key that would provide a searchable way to scan the raw logs for example like a word "iistart" or ".aspx" etc...  I am aware you can use the parsed meta keys, but what I am finding in an IR situation where I have had to load backdated logs, the parsing of the upload file is not as clean as I would like it.  Most of the data is still in the "Raw Log".  

How do we do this now:  Right now I will combine the log files into 3-4M log sessions and then use the "search bar" in the legacy investigate to scan the logs and get the results.  

Problem:  This is not efficient and just takes forever.   

Actual Question:  So is there a way I could create a custom META that maps the whole uploaded log file to a new key "RAW LOG META" like we map other key?  Anyone know how to do this?  The upload of the file on the decoder only really gives you two options, map with filename or map without filename.

[JoshRandall]

@ChrisIchelson 

The root of what you're asking for (indexing entire log messages) is not a scalable option, mostly due to the cost of the SSDs that would be required to support it, but also due to the performance cost it would incur.

I think what you're looking for is the Log Tokenizer engine to create word metas from your particular set of logs (you want to scroll down to the "Log Tokenizer Configuration Parameters" section to see all the config options): https://community.rsa.com/t5/rsa-netwitness-platform-online/log-decoder-service-configuration-parameters/ta-p/588303

By default, this option is set to only scan unknown logs --> logs that are not identified, matched with, and parsed by any available log parser. You can change this in the "token.device.types" option by adding the device.type value of your logs. You will then get word metas created from these logs, and can use the same "search bar" process you're already leveraging to run free text searches without needing to index the entire raw log.

A couple additional notes/docs on this topic:

• https://community.rsa.com/t5/rsa-netwitness-platform-blog/new-text-indexing-features-in-rsa-netwitness-v11-1/ba-p/518122
• https://community.rsa.com/t5/rsa-netwitness-platform-online/core-database-tuning-guide-index-customization/ta-p/569253 (the ngrams section)