How to Forward Syslog from a Remote Log Collector to another destination

[DavidWaugh1]

(I've only managed to get this method working with UDP)

Customers often want to forward syslog traffic that is being sent to Netwitness for Logs to another syslog destination, so that the syslog traffic is processed by both Netwitness and the other syslog receiver.

There are many ways that this can be achieved (such as Configure Syslog Forwarding to Destination - RSA Security Analytics Documentation ) but here is another method.

1) On the Remote Log Collector go to Event Sources -> Syslog and Config.

2) Disable the current syslog sources. These will typically be upd514 and tcp514. To disable edit each source and untick the Enabled box.

3) Set up new TCP and UDP listeners on a different port. In this case we use UDP 5514 and TCP 5514.

4) Stop and start the Syslog Collection Method

Make sure that in explore view under logcollection/syslog/eventsources/syslog-tcp and syslog udp that you set the following headers to true. This will ensure that when security analytics processes the message the original IP is seen.

5) Confirm with netstat -na |grep -i 514 that the system is only listening on your new ports 5514

 netstat -na |grep -i 514
tcp 0 0 0.0.0.0:5514 0.0.0.0:* LISTEN
tcp 0 0 192.168.123.242:41566 192.168.123.27:514 ESTABLISHED
tcp 1 0 192.168.123.242:51282 192.168.123.27:514 CLOSE_WAIT
tcp 0 0 :::5514 :::* LISTEN
udp 0 0 0.0.0.0:5514 0.0.0.0:*
udp 0 0 127.0.0.1:50514 0.0.0.0:*
udp 0 0 :::5514 :::*

6) Edit the file /etc/rsyslog.conf

Uncomment the following lines:

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
And add the line

$PreserveFQDN on

7) At the end of the file add the following lines:

*.* @@192.168.123.27:514

OR

*.* @192.168.123.27:514

This will forward all TCP traffic to 192.168.123.27 port 514 or all UDP traffic to 192.168.123.27

Also add one of the following lines

*.* @@localhost:5514

OR

*.* @localhost:514

This will forward all syslog traffic to our log collector for processing. The difference is that if @ is used the traffic gets forwarded as UDP syslog 514. If @@ is used the traffic gets forwarded as TCP. Forwarding as TCP is better as large syslog messages can be transferred and TCP has more robust packet delivery than UDP.

7) Restart the rsyslog service with service rsyslog restart

[DavidWaugh1]

I could only get this working if I forwarded the local messages as UDP so the line was 

*.* @192.168.123.27:514

I think this is due to the version of rsyslog that we are using on our system. The version by default is 

[origin software="rsyslogd" swVersion="8.4.1" x-pid="24970" x-info="http://www.rsyslog.com"]

I tried with the lastest version (see RHEL/CENTOS RPM Repository ) but this did not make a difference. I dont think that rsyslog liked forwarding traffic to the localhost.

[DavidWaugh1]

I don't know if anyone else can get this working like this. It seems that when I redirect syslog packets to itself I get the following in my /var/log/messages on the logcollector

Sep 23 15:17:08 VLC2 NwLogCollector[12559]: [Network] [failure] Unable to deserialize uint32, length is invalid (0) - 0x7f9,0e4,ee3,230
Sep 23 15:17:08 VLC2 NwLogCollector[12559]: [Network] [failure] Unable to deserialize uint32, length is invalid (0) - 0x7ff,f4c,07e,4d0
Sep 23 15:17:08 VLC2 NwLogCollector[12559]: [Network] [info] Connection 10644 (127.0.0.1) closing due to socket error (Invalid argument)

[JesseLyon]

I followed your directions and was able to set this up on all my VLCs to forward traffic to a 3rd party platform. It took a bit of tweaking to get it working as I liked, what I found worked best was using the following forwarding rules, sending TCP to the 3rd party, UDP to the local logcollector on the VLC itself.

*.* @@(remoteIP):514

*.* @localhost:5514

There were a few parsing errors, namely that the forward IP in log messages for some hosts always rendered as ::1 or 127.0.0.1, but the alias.host and other metakeys populated as they should. Disabling IPv6 seemed to help out parsing a bit, I'm not sure if that could have had an effect on how services bound to interfaces and didn't really investigate.

I also found that the /var/log/messages folder filled up very quickly on high volume VLCs (5k-10k EPS), so it was necessary to comment out the following line to prevent incoming syslog messages from being written to logs:

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.info;mail.none;authpriv.none;cron.none    /var/log/messages

Other than those issues, I was able to meet my objectives of feeding all ingested traffic VLCs received into both the local log collector and also to the 3rd party platform, with very little parsing issues in the NW platform, and high fidelity logs delivered to the 3rd party. 

[KEVINDIENST]

What if I wanted to keep the default listening ports on a Remote Log Collector and forward a duplicate of traffic we get via SCP for instance over to a UAT Remote Log Collector?

I'm attempting to push production log flows of a particular type to my UAT collector so I can assess a pipeline via Warehouse Connector. 

I know I'd have to change the listening ports on the UAT collector but wouldn't that still work to leave my PROD collectors unchanged?

Thanks! 

[DaveGlover]

There are a couple of ways to do this.  Personally I would use the 'Event Processor' in the VLC/RLC.

The following procedure will forward all events the VLC/RLC receives via Syslog regardless of how the events were received  (syslog/winrm/odbc,xx)

Here is the setup procedure on setting up the streaming event process from a log collector

This is NOT done on the log decoder ONLY the log collector

Open the Log Collector Explorer menu and find the "event Processor" section and right click and select properties

The Name can be anything you choose (without white spaces) 

Navigate to event-processors/GlobalStream/destinations/logdecoder/consumer/processors/tcpconnector/config/ connector/channel/tcp:

Ø

Finally Right-Click on event-processor/GlobalStream then click properties and select restart under the properties section: