Need help to parse JSON log

[MaximMarchenko]

Hello.

First of all, I've read this article and tried to do such thing, but no luck.

Logs are from MongoDB Community event source.

Log example:

"Jan 24 16:30:10 mongodb-linux-test-1 MONGODB-AUDIT: {"t":{"$date":"2022-01-24T16:30:01.081+06:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"192.168.1.100:58595","connectionId":37,"connectionCount":1}}"

I've tried to parse only these ("s":"I", "c":"NETWORK", "id":22943)

My log parser file is:

<?xml version="1.0" encoding="UTF-8"?>

<DEVICEMESSAGES

    name="mongodb_audit_custom"

    displayname="MONGODB CUSTOM"

    group="Database">

<VERSION

    xml="62"

    checksum="47a2bb0d3341b498eb15e4e72ebf1916"

    revision="143"

    device="2.0" />

<HEADER

    id1="0001"

    id2="0001"

    content="&lt;fld&gt; &lt;fld&gt; &lt;fld&gt; &lt;hhost&gt; &lt;messageid&gt;: &lt;!payload&gt;"/>

<MESSAGE

        id1="MONGODB-AUDIT"

        id2="MONGODB-AUDIT"

        content="&lt;logstash_json_payload&gt;" />

<VARTYPE name="logstash_json_payload" dataType="FileBeatsEvent"/>

<DataType name="ElasticCommonSchemaSubset" format="JSON">

   <Capture key="/s" meta="severity" />

   <Capture key="/c" meta="ec_subject" />

   <Capture key="/id" meta="reference_id" />

</DataType>

After (re-)deployment I've get such result(message id is OK, but no metas of severity, reference, subject):

Please, explain what is wrong with my parser file.

[MaximMarchenko · Accepted solution]

I've managed to find the error by myself:

<VARTYPE name="logstash_json_payload" dataType="FileBeatsEvent"/>

...

<DataType name="ElasticCommonSchemaSubset" format="JSON">

...

These name must be the same:

<VARTYPE name="logstash_json_payload" dataType="FileBeatsEvent"/>

...

<DataType name="FileBeatsEvent" format="JSON">

...

The result is: I get all my metas, described in <Capture ...>.

So somebody in RSA must fix this article .

Also in sections:

<VARTYPE name="logstash_json_payload" dataType=FileBeatsEvent"/>

<DataType name="InternetTime" dateTime="%W-%M-%DT%H:%T:%S.%V%E" />

<DataType name="CollectionTime" type="InternetTime" meta="lc.ctime"/>

<DataType name="ElasticCommonSchemaSubset" format="JSON">
   <Capture key="/@timestamp">     type="CollectionTime" />
   <Capture key="/host/hostname">  meta="alias.host" />
   <Capture key="/host/id">        meta="hardware.id" />
   <Capture key="/host/os/name">   meta="OS" />
</DataType>

[NamYongCho]

If the message is logstash/JSON, there is an easier way.
From v11.6, you can utilize Logstash/Beats plugin on Log Collector and use JSON Mappings on Log Parser Rules configuration.
Please, refer the attached ppt deck as an example.
Instead of writing a new parser, you can do same thing on web UI.

[anusha_rampure]

Hi,

If we make use of json mapping feature for logstash logs, how will it identify the header? As we will be using the header while mapping?

[NamYongCho]

For the JSON mapping, there is no concept of headers and messages.

As JSON is structured data, you only need to define hierachical path for each values.