Logs deduplication

[NikolayKlender]

Hi! Is there any way to deduplicate events? Say we have two firewalls (intrnal and external) and if some host tries to connect to internet site we will have two log records with same ip.src, ip.dst, ip.dstport. So It will be good to have ability to deduplicate logs in following way: 

1) define deduplication key like set of metas

2) define time period during which logs will be throtled in case of dedup key is equal

So I am looking some thing like logstash throttle filter.

I think it could be done with help of lua parser but there is question about thread safety.

[DaveGlover]

Netwitness does not Dedup logs.  From an investigative perspective this makes it much harder with hunting and finding true times of events.  

Also no parser has the ability to remove meta data.  So even it you tried to create a lua parser to remove duplicate data it would not function.  There is no remove functions built into the Database.  This is to prevent deletion of data in an event to cover up ones tracks