make API job "logstats" configurable in logdecoder appliance

[rsaeng]

Hello

logdecoder "logStats" API job (Retrieve count and last receive times for log devices, sources and forwarders) is interesting, but not modifiable enough.

curl https://______:50102/decoder?msg=logStats&force-content-type=text/plain

device:vontu_monitor
 forwarder:nwlogtest
 source:22.1.88.24
 discoveredType:vontu_monitor
 count:622950
 lastSeenTime:2023-May-26 07:05:20
 lastUpdatedTime:2023-Jun-06 07:41:48

it would be useful to be able to define the metas used in the job. For example, on our platform, source ip of a device isn't device.ip but another custom meta. so we didn't use this feature.

Could you make it possible to configure the metas used in this job in explore view ?

thanks.

[Salma-Soliman]

Hello @RaviAdireddi,

Can you please check the above idea and advise if it could be taken into consideration as RFE ?

Thank you 

Salma Soliman

[RaviAdireddi]

source, count, lastSeenTime and others are named variables and not meta.  Customer should be able to map with to whatever field they expect with a small program that makes a REST call and maps the fields as per their expectation.  

[Salma-Soliman]

Thanks @RaviAdireddi for your feedback.
Is it possible to share with us a guide or example to follow the above ? 

[rsaeng]

Hello @RaviAdireddi , 

I misspoke. My request is to be able to modify the meta used in the logdecoder source code for the "source" field. 

"logStats" job currently uses these metas for these fields:

meta used in job	field output
device.type	device
lc.cid	forwarder
device.ip	source

I would need to modify the "source" field. For example, a new parameter in explore view (logdecoder) for the logstats job:

/decoder/config/logs.stats.enabled (Log Stats Enabled) = true
/decoder/config/logs.stats.prune.interval (Log Stats Prune Interval) = 43200
=> NEW /decoder/config/logs.stats.source.meta (meta used to retrieve source IP) = ctx.orig.ip

To use this meta for the job counter. It can be very useful for this collection job.

do you understand my request?

[RaviAdireddi]

Log stats don’t come from meta. They come from the context of delivery. The key names you see aren’t meta names. They are context values that form a tuple which defines the source and have no connection to meta.

This context can be overridden…

The forwarder can include the source address, so it’s used instead of the connecting (sending) client’s address.

This is done w/ a NetWitness annotated RFC-5424 header or a legacy 5 bracketed envision header (z-connector).

The formats for the two header types (the forwarder would default to the sending client’s address):

<${syslog_pri}>1 - ${souce_address} ${device_name} - - [lc@36807 lc.ctime="${ms_collection_time}"] ${the_log}

[${device_name}][][${souce_address}][${ms_collection_time}][] ${the_log}

Formats including a specific forwarder:

<${syslog_pri}>1 - ${souce_address} ${device_name} - - [lc@36807 lc.ctime="${ms_collection_time}" lc.cid="${forwarder}"] ${the_log}

[${device_name}][${forwarder}][${souce_address}][${ms_collection_time}][] ${the_log}

Examples (apache):

<13>1 - 1.2.3.4 apache - - [lc@36807 lc.ctime="1310902593292"] This is the log
<13>1 - hostname apache - - [lc@36807 lc.ctime="1310902593292"] This is the log
[apache][][1.2.3.4][1310902593292][] This is the log

[apache][][hostname][1310902593292][] This is the log

Examples specific to the device below in thread w/ explicit forwarder (vontu_monitor):

<13>1 – 1.2.3.4 vontu_monitor - - [lc@36807 lc.ctime="1310902593292" lc.cid="nwlogtest"] This is the log
[vontu_monitor][nwlogtest][1.2.3.4][1310902593292][] This is the log

[rsaeng]

hello Ravi,

"device" logstats field isn't included in the context of delivery (by a VLC). It's extracted after parsing on a logdecoder.

In the C code of the 'logstats' module, a section retrieves the device.type meta on each log for the "device" field ? So you can extract a custom meta for the "source" field 

[JacobVos]

The device is discovered or provided in the delivery context (NetWitness log headers). It's not extracted, unless it's a CEF log.

The same is true for the other two values, source and forwarder, and they can be set via delivery context. When context is absent, the source is derived from the address of the connecting client that sent the log and there is no forwarder. Often the "client" is the Local Log Collector that cohabits the Log Decoder Appliance, so Log Collectors set the original source in the headers prefixed to each log sent to the Log Decoder Service to provide this context. The Log Collector records itself as the forwarder.

If the source is present in context, but the forwarder is absent, then the connecting address becomes the forwarder. If both are present, the connecting address is ignored. For example, the Local Log Collector's connecting address is always localhost, which is not very useful, so Log Collector always sets the source and forwarder in the NetWitness headers it prefixes to each log.

The source does not map from the meta device.ip, but instead the same "source" information (discovered or through context) is saved to meta, which could also be device.ipv6 or device.host. The same is done for the "forwarder".

The source can be anything you put in the header. If it's not identified as an IPv4 or IPv6 value it's treated as a hostname, which can be anything as it's simply a text value.

Nearly all forwarders have a means to customize the message format and have access variables (like the origins IP) to provide context for what is being passed along.

The formats outlined in Ravi's last message should allow for the accurate values to be utilized as the source and optionally the forwarder.