Indexing Log Source Event Time and Lua Parser Callbacks?

[KEVINDIENST]

Wondering if anyone on here has setup a method to index and/or parser the native log source timestamps?

We have problems in that some log sources may get backed up when our log decoders are either down or we do a long upgrade and we haven't found a solution for our analysts when the SA ingestion time and original log source time don't align. It is very frustrating for them when they can't find an event and are under stress to get incidents resolved. 

It makes it worse that we have log sources logging in different time zones and owners not being super excited about adjusting everything into UTC and verifying they have accurate NTP settings. 

I know the log decoders index the ingestion time (only thing indexed on the log decoders) and that attempting to index the raw event time from the source will have a huge performance hit, in addition to adding the burden of having to update parsers for all the log sources we support since they may all format time a little differently. 

I spoke with some RSA resources about doing this and they mentioned potentially limiting the indexing to a few critical/problematic log source types and using Lua to parse the time into small buckets (i.e. into minutes or 5-10 minute buckets instead of down to the millisecond for instance). 

Anybody else have ideas or can extrapolate on using Lua to accomplish this?

Thank you! XDudADtNBir2yWTvsAvML3S8A4BmZKPAABc5iGXZy0M=N6EPgVyqjIeX4VQaO87WEl41adMR7kmOen9eAKHNIG0=Lb4PID2jPLBSBnAF2o6daHw6uLL4Qn1o12RToTXYDMM=

[KEVINDIENST]

We have remote collectors collecting data first, then they push upstream to log decoders, but my understanding is that we cannot set up natively a way to round-robin upstream log decoders. 

If you know of a way for me to do this out of the box, so if a log decoder outage happens I can still push logs upstream to an alternative log decoder automatically, I'd love to hear more.