2014-09-23 03:03 AM
Hello,
After update from 10.3.4 to 10.4 I receive some problem. My Log Collector not work. In first time I receive many like:
Sep 23 09:36:00 SA-LogDecoder nw[2196]: [AMQPClientBase] [failure] An error occurred creating an AMQP channel: a socket error occurred
Sep 23 09:36:00 SA-LogDecoder nw[2196]: [LogdecoderProcessor] [failure] [queue.odbc] [idle] Failed during getWork: a socket error occurred
I can see only logs collect via log decoder (protocol Syslog). Then I try find solution at forum and find action rekey (). I'm also reinstall nwlogcollector, rabbit-mq server. Now I can access to Log Collector (Service x.x.x.x host LogCollector is unreachable).
2014-09-23 04:35 AM
How I guess problem in rabbitmq-server. NwLogCollector not start automatically.
BOOT FAILED
===========
Error description:
{could_not_start,nw_admin,
{{shutdown,
{failed_to_start_child,nw_admin_worker,
{{badmatch,false},
[{nw_admin_worker,init,1,[]},
{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}}},
{nw_admin,start,[normal,[]]}}}
=INFO REPORT==== 23-Sep-2014::12:30:19 ===
node : rabbit@SA-LogDecoder
home dir : /var/lib/rabbitmq
config file(s) : (none)
cookie hash : Fb/V18D0IfbwgiH0zgGgvw==
log : /var/log/rabbitmq/rabbit@SA-LogDecoder.log
sasl log : /var/log/rabbitmq/rabbit@SA-LogDecoder-sasl.log
database dir : /var/lib/rabbitmq/mnesia/rabbit@SA-LogDecoder
=INFO REPORT==== 23-Sep-2014::12:30:22 ===
Limiting to approx 924 file handles (829 sockets)
=INFO REPORT==== 23-Sep-2014::12:30:22 ===
Memory limit set to 3149MB of 7872MB total.
=INFO REPORT==== 23-Sep-2014::12:30:22 ===
Disk free limit set to 50MB
=INFO REPORT==== 23-Sep-2014::12:30:22 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index
=INFO REPORT==== 23-Sep-2014::12:30:22 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index
=WARNING REPORT==== 23-Sep-2014::12:30:22 ===
msg_store_persistent: rebuilding indices from scratch
=INFO REPORT==== 23-Sep-2014::12:30:22 ===
Adding vhost '/'
=INFO REPORT==== 23-Sep-2014::12:30:22 ===
Creating user 'guest'
=INFO REPORT==== 23-Sep-2014::12:30:22 ===
Setting user tags for user 'guest' to [administrator]
=INFO REPORT==== 23-Sep-2014::12:30:22 ===
Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
=INFO REPORT==== 23-Sep-2014::12:30:22 ===
started TCP Listener on [::]:5672
=INFO REPORT==== 23-Sep-2014::12:30:23 ===
Management plugin started. Port: 15672
=INFO REPORT==== 23-Sep-2014::12:30:23 ===
Statistics database started.
=INFO REPORT==== 23-Sep-2014::12:30:23 ===
nw_admin_worker:init: [{included_applications,[]}]
=INFO REPORT==== 23-Sep-2014::12:30:23 ===
nw_admin initialized.
=INFO REPORT==== 23-Sep-2014::12:30:23 ===
stopped TCP Listener on [::]:5672
=INFO REPORT==== 23-Sep-2014::12:30:23 ===
Error description:
{could_not_start,nw_admin,
{{shutdown,
{failed_to_start_child,nw_admin_worker,
{{badmatch,false},
[{nw_admin_worker,init,1,[]},
{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}}},
{nw_admin,start,[normal,[]]}}}
{"init terminating in do_boot",{rabbit,failure_during_boot,{could_not_start,nw_admin,{{shutdown,{failed_to_start_child,nw_admin_worker,{{badmatch,false},[{nw_admin_worker,init,1,[]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}},{nw_admin,start,[normal,[]]}}}}}
[60G[ [0;31mFAILED [0;39m]
=CRASH REPORT==== 23-Sep-2014::12:30:23 ===
crasher:
initial call: nw_admin_worker:init/1
pid: <0.361.0>
registered_name: []
exception exit: {{badmatch,false},
[{nw_admin_worker,init,1,[]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}
in function gen_server:init_it/6 (gen_server.erl, line 328)
ancestors: [nw_admin_sup,<0.358.0>]
messages: []
links: [<0.359.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 1598
stack_size: 27
reductions: 4311
neighbours:
=SUPERVISOR REPORT==== 23-Sep-2014::12:30:23 ===
Supervisor: {local,nw_admin_sup}
Context: start_error
Reason: {{badmatch,false},
[{nw_admin_worker,init,1,[]},
{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}
Offender: [{pid,undefined},
{name,nw_admin_worker},
{mfargs,
{nw_admin_worker,start_link,
[[{included_applications,[]}]]}},
{restart_type,permanent},
{shutdown,10000},
{child_type,worker}]
=CRASH REPORT==== 23-Sep-2014::12:30:23 ===
crasher:
initial call: application_master:init/4
pid: <0.357.0>
registered_name: []
exception exit: {{shutdown,
{failed_to_start_child,nw_admin_worker,
{{badmatch,false},
[{nw_admin_worker,init,1,[]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,304}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}}},
{nw_admin,start,[normal,[]]}}
in function application_master:init/4 (application_master.erl, line 133)
ancestors: [<0.356.0>]
messages: [{'EXIT',<0.358.0>,normal}]
links: [<0.356.0>,<0.7.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 376
stack_size: 27
reductions: 124
neighbours:
2014-09-23 10:03 AM
How I can understand - I lost my keys in folder /etc/netwitness/ng/rabbitmq/ssl/keys at LogCollector/Decoder. How I can generate new keys?
2014-09-24 02:34 AM
I am seeing the same issue. you mention that you upgraded to 10.4 and did you reinstall nwlogcollector and rabbitmq ? did it resolve the issue ?
2014-09-24 02:55 AM
I make same actions like reinstall erlang*, rabbitmq-server, nwlogcollector... nothing happend.
But I have same progress...
1. service rabbitmq-server stop, stop nwlogcollector
2. rm -rf /etc/netwitness/ng/rabbitmq/legacy-ssl
3. cp /opt/netwitness/etc/rabbitmq-base-config /etc/rabbitmq/config/rabbitmq.config
4. ln -s /etc/netwitness/ng/rabbitmq/config/rabbitmq.config /etc/rabbitmq/config/rabbitmq.config
5. rm /etc/rabbitmq/ssl/truststore.pem
6. service rabbitmq-server start, start nwlogcollector
7. Administration --> Devices --> Select Log Collector --> Explore --> event-broker --> ssl --> (Right-Click) Properties --> (from drop down) rekey
After this actions I reboot log collector and links in my folder /etc/netwitness/ng/rabbitmq/ssl recovered. But this links to:
cert.pem /etc/rabbitmq/ssl/server/cert.pem
privkey.pem /etc/rabbitmq/ssl/server/key.pem
This files and folder "server" not exist at my log collector. I look at similar files at SA Server, this file - links to
/var/lib/puppet/ssl/certs/id.pem
/var/lib/puppet/ssl/private_keys/id.pem
I make similar links, but this not happend 😞
2014-09-25 01:09 PM
I am also experiencing this problem after the 10.4 update.
2014-09-25 11:08 PM
Please also check the host file which is etc/host and make sure you see a loop back address.I had same issue and seems my host file got corrupted. and fixing the host file thing are normal, but see a different error now.
Keyur
2014-09-26 02:06 AM
Hosts file correct. I guess problem in bad links in folder /etc/netwitness/ng/rabbitmq/ssl/keys/:
cert.pem -> /etc/rabbitmq/ssl/server/cert.pem
privkey.pem -> /etc/rabbitmq/ssl/server/key.pem
This files not exist. All ssl keys from previously version go to folder /etc/netwitness/ng/rabbitmq/legacy-ssl/keys/, but if we look at View/Explorer we see links to /etc/netwitness/ng/rabbitmq/ssl/keys/. I try cope files from folder ../legacy-ssl/keys/ to ../ssl/keys/ - this not happend. I guess new keys - puppet keys like in SA Server. I make similar links, but this not happed. Need contact to RSA Support... or need contact with customer who correctly update to 10.4.
2014-10-01 12:12 PM
Anyone figure this out? I contacted support today but curious.
2014-10-02 01:48 AM
Please, share answer RSA Support here.
I not have more time to wait solution. I reimage my demo instance. But I not "Enable" at tab "Administrator -> Appliance" my Log Decoder/Collector before I try solve problem. After "Enable" - this not helped me, maybe because I done more same operations.. Maybe If I done all step by step from update guide - update be success.