secure 2.3 GB file under /var/log

[DeepanshuSood1]

Hi,

I have one Virtual log collector running on v 10.4.0.2 and what i observed in that VLC, that there is one file which is being created under /var/log/ and the file name is secure and it have a size of more 2.3 GB, which i have deleted many times, but after some time it again gets appeared at the same location which is slowing down the vlc.

And below are some line of logs which are in the file if if i look into it.

Aug  9 06:58:14 NCORP-VLC-01 sshd[14558]: debug1: user sftp matched group list uploads at line 159

Aug  9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: match found

Aug  9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: reprocess config:160 setting ChrootDirectory /var/netwitness/logcollector/upload_chroot

Aug  9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: reprocess config:161 setting X11Forwarding no

Aug  9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: reprocess config:162 setting AllowTcpForwarding no

Aug  9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: reprocess config:163 setting PasswordAuthentication no

Aug  9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: auth_shadow_acctexpired: today 17022 sp_expire -1 days left -17023

Aug  9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: account expiration disabled

Aug  9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1

Aug  9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: mm_request_send entering: type 8

Aug  9 06:58:14 NCORP-VLC-01 sshd[14558]: debug2: monitor_read: 7 used once, disabling now

Aug  9 06:58:14 NCORP-VLC-01 sshd[14558]: debug3: mm_request_receive entering

Aug  9 06:58:14 NCORP-VLC-01 sshd[14560]: debug2: input_userauth_request: setting up authctxt for sftp

Aug  9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_inform_authserv entering

Aug  9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_request_send entering: type 3

Aug  9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_inform_authrole entering

Aug  9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_request_send entering: type 4

Aug  9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_auth2_read_banner entering

Aug  9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_request_send entering: type 9

Aug  9 06:58:14 NCORP-VLC-01 sshd[14560]: debug3: mm_request_receive_expect entering: type

So I am just wondering that why the logs are being generating and how to get rid from them.

Kindly advise. Thanks.

Regards,

Deepanshu Sood.

[DeepanshuSood1]

Yes the secure file is still there and it's keep on modifying.

The below are some sample logs..

Aug  8 06:19:01 NCORP-VLC-01 sshd[4616]: debug2: kex_parse_kexinit:

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: reserved 0

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: ssh-rsa,ssh-dss

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit:

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit:

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: first_kex_follows 0

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: kex_parse_kexinit: reserved 0

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug2: mac_setup: found hmac-md5

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug1: kex: client->server aes128-ctr hmac-md5 none

Aug  8 06:19:01 NCORP-VLC-01 sshd[4617]: debug3: mm_request_send entering: type 78

Aug  8 06:19:01 NCORP-VLC-01 sshd[4612]: debug3: monitor_read: checking request 78

Aug  8 06:19:01 NCORP-VLC-01 sshd[4616]: debug2: kex_parse_kexinit:

Aug  8 06:19:01 NCORP-VLC-01 sshd[4616]: debug2: kex_parse_kexinit: first_kex_follows 0

Aug  8 06:19:01 NCORP-VLC-01 sshd[4616]: debug2: kex_parse_kexinit: reserved 0

Aug  8 06:19:01 NCORP-VLC-01 sshd[4616]: debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

[KhaledGamal]

Hi Sood,

From the logs you sent it shows that SSH is still having debug logs. From the below snippet in my lab:-

[root@sa ssh]# cat /var/log/secure | grep -i debug

[root@sa ssh]#

This shows that there is no debug logs by default. The changes that were done on your VLC has to be found out to revert it back as it was.

Check both sshd_config and ssh_config under /etc/ssh and check if there is any option that was added to increase the log level.

You can compare the files with another appliance that doesn't have DEBUG logs in the secure files to determine what changes were done.

Best regards

Khaled

[DeepanshuSood1]

Hi Khaled,

I found one thing in the sshd_config file related to Logging Level which is 3.
I have sent an email to you of the files.

Regards,

Deepanshu Sood

[KhaledGamal]

Hi Sood,

I checked the files sent and in the sshd_config there is the below line:-

LogLevel DEBUG3

This has to be changed to:-

LogLevel INFO

After that you will have to restart the sshd service for the new config to take effect.

PS:- When restarting do a restart not a stop then start as if you do a stop, you will lose access to the SSH session.

Best regards

Khaled

[DeepanshuSood1]

Khaled, after the change in the file and restarting the service. The file is still modifying.

[KhaledGamal]

Send me the output of the below commands:-

1- date

2- tail -10 /var/log/secure

3- cat /etc/ssh/sshd_config | grep -i debug

4- cat /etc/ssh/ssh_config | grep -i debug

5- cat /etc/sysconfig/sshd | grep -i debug

6- cat /etc/ssh/sshd_config | grep -i log

7- cat /etc/ssh/ssh_config | grep -i log

8- cat /etc/sysconfig/sshd | grep -i log

[DeepanshuSood1]

output sent.

Regards,

Deepanshu Sood.

[KhaledGamal]

Hi Sood, I checked the output and all is good now.

Note that before all the logs you sent in your last post here in Link is in the same second "Aug  8 06:19:01" because all were debug logs.

Now in the mail you sent me, in a 3 minute interval there were only 10 logs. This is normal and that is the way the log file should be.

You could delete the file if it is already big now from the debug logs and from now it won't log debug logs and it won't get big in size as it did before.

Hope this helps!

Best regards

Khaled

[DeepanshuSood1]

You're rocking Khaled.  Big thanks.

But one thing I'm amazed that how the changes are made in the files?

Any guess?

Regards,

Deepanshu Sood.

[KhaledGamal]

Hi Sood,

Anytime

Any SSH session that occurs gets logged in including any protocol that uses SSH for example sFTP. You can check the IPs that are logged in the file to understand exactly what sessions are being logged.

Best regards

Khaled