Use Stack-Trace script to analyze core file for NetWitness core services

[JohnWu1]

Sometimes NetWitness core service decoders and concentrators will crash and resulting a core.xxx file in one of the packetdb or metadb directories. As a result of the crash, affected decoder or concentrator service sometimes will NOT start properly anymore.

To assist RSA CE or development team to resolve the issue and find the root cause, one can run the attached RSA Stack-Trace script on the affected decoder or concentrator (ssh as root) to generate a text file for RSA CE or development team to review. This is very useful since the core.xxx file is usually VERY LARGE but Stack-Trace script output text file is normally very small, and RSA CE or development team can usually detect the issue only from this script output file.

Below is an example of how to run Stack-Trace script on a packet decoder (needs to ssh into packet decoder as root user):

1) a core.30441 file (61GB in size!) is generated on the decoder /var/netwitness/decoder/packetdb directory and decoder service won’t start properly afterwards:

[root@packedecoder packetdb]# ls -ltr core*

-rw-------. 1 root root 61866070016 Sep 17 20:33 core.30441

[root@packedecoder packetdb]#

2) scp Stack-Trace script to decoder and make executable:

[root@packedecoder ~]# ll

-rw-r--r--. 1 root root   2177 Sep 23 15:50 Stack-Trace

[root@packedecoder~]# chmod +x Stack-Trace

[root@packedecoder ~]# status nwdecoder

nwdecoder start/running, process 9528

[root@packedecoder ~]#

3) run Stack-Trace script against the core.30441 file to generate core.result (text) file:

Note: one can run Stack-Trace script with decoder service running or stopped (above showed decoder service is still running):

[root@packedecoder ~]# ./Stack-Trace /usr/sbin/NwDecoder /var/netwitness/decoder/packetdb/core.30441 > core.result

No /proc directory: '/proc/30441'

root@packedecoder ~]# ls -l core*

-rw-r--r--. 1 root root 44798 Sep 23 15:51 core.result

[root@packedecoder ~]# cat core.result

Using file : /usr/sbin/NwDecoder

########################################

Process: /usr/sbin/NwDecoder, pid:

Corefile: /var/netwitness/decoder/packetdb/core.30441

****Process Stack Trace:****

…..

[root@packedecoder ~]#

4) Send above core.result text file for RSA CE or development team for review.

Note one can also run below gdb command against the same core.30441 file to generate a text file but that text file in general will NOT contain many useful information since gdb requires RSA CE or development teams to provide symbol tables first before it can be run successfully, so run Stack-Trace script against the core.30441 file is in general preferred:

[root@packetdecoder packetdb]# gdb /usr/sbin/NwDecoder core.30441 > gdb-core.result

 Ctrl+C

[root@packetdecoder packetdb]# cat gdb-core.result

…

Reading symbols from /usr/lib64/gconv/UTF-32.so...(no debugging symbols found)...done.

Loaded symbols for /usr/lib64/gconv/UTF-32.so

Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.

Loaded symbols for /lib64/libnss_files.so.2

Core was generated by `/usr/sbin/NwDecoder --stopwhenready'.

Program terminated with signal 11, Segmentation fault.

#0 0x00007f9a892a35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

Missing separate debuginfos, use: debuginfo-install

[root@packetdecoder packetdb]# 

I have also attached both core.result.txt and gdb-core.result.txt files from above sample run so one can see the different information each generated.

gdb-core.result.txt.zip

core.result.txt.zip

Stack-Trace.zip