nginx logs to RSA netwitness

[azimgambarli]

Dears,

Has anybody tried to monitor nginx web server using RSA Netwitness? If so please share the config sample to forward logs to Netwitness from linux web server?  I tried apache log source configuration guide with no luck.

Any help appreciated.

[EricaChalfin]

azim gambarli‌,

I've moved your question to the RSA NetWitness Platform" data-type="space space where it will be seen by the product's support engineers, other customers and partners.  Please bookmark this page and use it when you have product-specific questions.

Alternatively, from the RSA Customer Support page, click on Ask A Question on the blue navigation bar and choose Ask A Product Related Question.  From there, scroll to RSA NetWitness Platform" data-type="space‌ and click Ask A Question.  That way your question will appear in the correct space.

Regards,

Erica

[RafaelSampaio]

Hi!

Did you try sending logging messages using Nginx instead of syslogd? You can check the configuration here https://docs.nginx.com/nginx/admin-guide/monitoring/logging/

[azimgambarli]

Dear Rafael,

I used the link to setup syslog via nginx. But experienced the same issue as it was with rsyslog. The logs coming to RSA Netwitness with extra "syslog header" which repeats Date and contains Tag and hostname, despite of their local storage format. These logs parsed incorrectly.

Any additional suggestions?

[RafaelSampaio]

Hi Azim!

Which parser is processing the Nginx logs in your environment?

One approach would be to create a new parser according to your log format. You may use try modifying the log format to a key-value config so you could use Log Parser Rules (Log Parser Customize: Add a Log Parser Rule)

Another approach could be formating messages in the same way NetWitness parses Apache logs if you can add the same headers in the message, but I'm not fully aware of the options available in log format in Nginx. (Apache HTTP Server Event Source Configuration Guide)

"%h %l %u %t \"%m \"%V\" \"%U\" \"%q\" %H\" %>s %b \"% {Referer}i\" \"%{User-Agent}i\" \"%{Cookie}i\""

%h - the IP address of the client (remote host)
%l - remote logname, usually a dash
%u - remote user if the request was authenticated
%t - time the request was received
\" - quotes
%m - request method
\"%V\" - server name with quotes
\"%U\" - URL path with quotes
\"%q\" - query string with quotes
%H - protocol
\" - quotes
%>s - HTTP status code
%b - size of response in bytes
\"%{Referer}i\" - Referer for all codes with quotes
\"%{User-Agent}i\" - agent (browser) with quotes
\"%{Cookie}i\" - entire cookie header with quotes

[azimgambarli]

Dear Rafael,

Everything has been set up as you broadly described and Apache HTTP Server Event Source Configuration Guide used for rsyslog and log format configuration.

In the access log file logs look the same as they should be and configured. But Rsyslog adds additional date and hostname tag, this brakes the way log parsed (yes it has been finally parsed) by apache parser:

May 29 15:58:30 localhost %APACHE- xxx.xxx.xxx.xxx - - [29/May/2020:15:58:29 +0400] "GET etc....

The fields important and required for adequate web server monitoring such as url, user-agent, referrer has been parsed incorrectly. The issue persists. 

[RafaelSampaio]

Hi Azim,

The hostname and timestamp are normal in Syslog messages and both are expected by the Apache parser. It can parser messages with and without those headers, so it should be working normally.
What is the device.type in your parsed events? Is there any "word" meta key?

[DaveGlover]

azim gambarli‌ Is it possible to post a log sample?  Or send to me directly?

[azimgambarli]

Rafael Sampaio wrote:

Hi Azim,

The hostname and timestamp are normal in Syslog messages and both are expected by the Apache parser. It can parser messages with and without those headers, so it should be working normally.
What is the device.type in your parsed events? Is there any "word" meta key?

 Dear Rafel,

As you have mentioned the messages can be parsed without those header. But the issue is that logs arrive with those additional headers, and parsed incorrectly.