2016-08-30 02:54 PM
I'm sure most of you are now starting to regularly deploy the new Series 5 Hardware. Many of us have used the NwArrayConfig script (2.1 / 2.2) versions in the past, but you may encounter issues when running these older scripts on Series 5 Gear with "Multiple DACs on Concentrators specifically" It has been suggested to me to use the new script located in "/opt/rsa/saTools". You can also acquire this also by downloading the following rpm;
rsa-sa-tools-10.6.1.1-118.4.eb0e5a2.el6.noarch.rpm
DO NOT use the NwArrayCfg script in the opt/rsa/saTools directory from any release earlier than the release listed above. It will incorrectly configure the first two drives in each Packet Decoder DAC into a RAID 0 Configuration.
--------------------------------------------------------------------
Hello Everyone,
This Article contains the error(s) and the workaround for this particular issue.
I encountered the issue below while deploying some S5 gear for a customer. The deployment consisted of (2) S5 Decoders with (5) DACs each, and (2) S5 Concentrators with (4) DACs each. The gear was re-imaged/down-revved to 10.4.0.2.J because the customer was running at version 10.4.1.4 at the time of the installation. After the gear was spun up and IP’d the ArrayCfg script was ran (2.2) version on each of the Decoders, and all (5) DACs on each initialized without issue, it was only the S5 Concentrators that had this issue.
I was able to successfully run the script the first time with the “- - action init “ option, which configured the first DAC correctly. When attempting to run the script a second time with the “ - -action add” option it failed and I got the following error message;
Failed!: The number of disk group sizes (3) is not valid for configuring a NextGen device. Please verify there are only 1 or 2 disk group sizes being presented to the appliance and rerun this script.
I ran the nwraidutil.pl script to check the status and saw the following; See in ORANGE that it appeared to have configured the second DAC, but when doing a df –Ph I could see that it indeed did not. Also toward the end of the nwraidutil.pl output it showed that I had physical disk issues. Also see in ORANGE.
--------------------------------------------------------------------
[root@sanitizedhostname ~]# nwraidutil.pl
RSA NetWitness RAID Utility Version 2012.09.07
Using /opt/MegaRAID/MegaCli/MegaCli64
-----------------------------
Logical Disk Information
-----------------------------
Physical Drive State Legend
--------------------------------------------------------------------
B Unconfigured(Bad) O Online
D Dedicated Hotspare and associated virtual drive number R Rebuild
E Hotspare prefers same enclosure S Solid-State Drive
F Foreign U Unconfigured(Good)
G Global hotspare X Offline
I Hotspare is revertible ! Failed
M Missing ? Unknown state
--------------------------------------------------------------------
NOTE: 'E' does not prohibit a hotspare from being used in another enclosure, it is merely a preference
Logical Drive State Legend
------------------------------------------------
D Degraded X Offline
O Optimal ! Failed
P Partially Degraded ? Unknown state
R Rebuild
------------------------------------------------
Adapter: 0 - PERC H730P Mini
Virtual Disk: 0 (O) - Found 2 of 2 {Raid Level 1, 931.0 GB, 128 KB Stripe Size, WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU}
PD: 0 Enclosure: 32 Slot: 0 (O) 931.512 GB SEAGATE ST91000640SS AS0B9XG944R5
PD: 1 Enclosure: 32 Slot: 1 (O) 931.512 GB SEAGATE ST91000640SS AS0B9XG944S5
Virtual Disk: 1 (O) - Found 2 of 2 {Raid Level 1, 1.818 TB, 128 KB Stripe Size, WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU}
PD: 0 Enclosure: 32 Slot: 2 (O) 1.819 TB SEAGATE ST2000NX0273 NS27S4607JZR
PD: 1 Enclosure: 32 Slot: 3 (O) 1.819 TB SEAGATE ST2000NX0273 NS27S4607HXX
Adapter: 1 - PERC H830 Adapter
Virtual Disk: 0 (O) - Found 8 of 8 {Raid Level 5, 25.466 TB, 128 KB Stripe Size, WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU}
PD: 0 Enclosure: 51 Slot: 6 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RA4L
PD: 1 Enclosure: 51 Slot: 7 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RAFY
PD: 2 Enclosure: 51 Slot: 8 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RAJ9
PD: 3 Enclosure: 51 Slot: 9 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RAWE
PD: 4 Enclosure: 51 Slot: 10 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RA73
PD: 5 Enclosure: 51 Slot: 11 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RANY
PD: 6 Enclosure: 51 Slot: 12 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8QVM2
PD: 7 Enclosure: 51 Slot: 13 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RAE5
Virtual Disk: 1 (O) - Found 6 of 6 {Raid Level 5, 928.75 GB, 128 KB Stripe Size, WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU}
PD: 0 Enclosure: 51 Slot: 0 (OS) 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZAAYA
PD: 1 Enclosure: 51 Slot: 1 (OS) 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZD5GA
PD: 2 Enclosure: 51 Slot: 2 (OS) 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZAL4A
PD: 3 Enclosure: 51 Slot: 3 (OS) 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ768A
PD: 4 Enclosure: 51 Slot: 4 (OS) 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZA6PA
PD: 5 Enclosure: 51 Slot: 5 (OS) 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZD02A
Virtual Disk: 2 (O) - Found 6 of 6 {Raid Level 5, 928.75 GB, 128 KB Stripe Size, WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU}
PD: 0 Enclosure: 52 Slot: 0 (OS) 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ6NRA
PD: 1 Enclosure: 52 Slot: 1 (OS) 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ6NUA
PD: 2 Enclosure: 52 Slot: 2 (OS) 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ6RBA
PD: 3 Enclosure: 52 Slot: 3 (OS) 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZAL9A
PD: 4 Enclosure: 52 Slot: 4 (OS) 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ66HA
PD: 5 Enclosure: 52 Slot: 5 (OS) 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ61EA
Virtual Disk: 3 (O) - Found 8 of 8 {Raid Level 5, 25.466 TB, 128 KB Stripe Size, WriteBack, ReadAdaptive, Cached, Write Cache OK if Bad BBU}
PD: 0 Enclosure: 52 Slot: 6 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8R9AH
PD: 1 Enclosure: 52 Slot: 7 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RBP9
PD: 2 Enclosure: 52 Slot: 8 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RA8Y
PD: 3 Enclosure: 52 Slot: 9 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8R963
PD: 4 Enclosure: 52 Slot: 10 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8R9BL
PD: 5 Enclosure: 52 Slot: 11 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RBKQ
PD: 6 Enclosure: 52 Slot: 12 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RAML
PD: 7 Enclosure: 52 Slot: 13 (O) 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8F3VV
No logical disk problems found.
------------------------------
Physical Disk Information
------------------------------
Physical Drive State Legend
--------------------------------------------------------------------
B Unconfigured(Bad) O Online
D Dedicated Hotspare and associated virtual drive number R Rebuild
E Hotspare prefers same enclosure S Solid-State Drive
F Foreign U Unconfigured(Good)
G Global hotspare X Offline
I Hotspare is revertible ! Failed
M Missing ? Unknown state
--------------------------------------------------------------------
NOTE: 'E' does not prohibit a hotspare from being used in another enclosure, it is merely a preference
Adapters found: 2
Adapter 0 (PERC H730P Mini) enclosures found: 1
Adapter 0 (PERC H730P Mini) enclosure 32 slots found: 4
Encl Slot State P.Fail.Count Raw Size Inquiry Data
32 0 (O) 0 931.512 GB SEAGATE ST91000640SS AS0B9XG944R5
32 1 (O) 0 931.512 GB SEAGATE ST91000640SS AS0B9XG944S5
32 2 (O) 0 1.819 TB SEAGATE ST2000NX0273 NS27S4607JZR
32 3 (O) 0 1.819 TB SEAGATE ST2000NX0273 NS27S4607HXX
Adapter 1 (PERC H830 Adapter) enclosures found: 4
Adapter 1 (PERC H830 Adapter) enclosure 51 slots found: 15
Encl Slot State P.Fail.Count Raw Size Inquiry Data
51 0 (OS) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZAAYA
51 1 (OS) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZD5GA
51 2 (OS) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZAL4A
51 3 (OS) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ768A
51 4 (OS) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZA6PA
51 5 (OS) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZD02A
51 6 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RA4L
51 7 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RAFY
51 8 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RAJ9
51 9 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RAWE
51 10 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RA73
51 11 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RANY
51 12 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8QVM2
51 13 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RAE5
51 14 (GEI) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8R9YY Hotspare Information
Adapter 1 (PERC H830 Adapter) enclosure 52 slots found: 15
Encl Slot State P.Fail.Count Raw Size Inquiry Data
52 0 (OS) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ6NRA
52 1 (OS) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ6NUA
52 2 (OS) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ6RBA
52 3 (OS) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZAL9A
52 4 (OS) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ66HA
52 5 (OS) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ61EA
52 6 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8R9AH
52 7 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RBP9
52 8 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RA8Y
52 9 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8R963
52 10 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8R9BL
52 11 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RBKQ
52 12 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RAML
52 13 (O) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8F3VV
52 14 (ID-3) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z829MB Hotspare Information
Adapter 1 (PERC H830 Adapter) enclosure 67 slots found: 15
Encl Slot State P.Fail.Count Raw Size Inquiry Data
67 0 (US) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ633A
67 1 (US) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZBN4A
67 2 (US) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZBJJA
67 3 (US) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZADUA
67 4 (US) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZBM6A
67 5 (US) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZD2SA
67 6 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8HK81
67 7 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8HKM7
67 8 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8HHVD
67 9 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z970N5
67 10 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8E3G7
67 11 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8R929
67 12 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RA6Q
67 13 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RBSY
67 14 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8GJ8P
Adapter 1 (PERC H830 Adapter) enclosure 79 slots found: 15
Encl Slot State P.Fail.Count Raw Size Inquiry Data
79 0 (US) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZA86A
79 1 (US) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZA9YA
79 2 (US) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ859A
79 3 (US) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ89RA
79 4 (US) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZAHKA
79 5 (US) 0 186.310 GB HITACHI HUSRL402 CLAR200C190XTVZ66ZA
79 6 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RA81
79 7 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8N9JZ
79 8 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8R9PL
79 9 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8R9ZY
79 10 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RA0S
79 11 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RBAD
79 12 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8R9AY
79 13 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8RANL
79 14 (U) 0 3.638 TB SEAGATE ST4000NXCLAR4000GS18Z1Z8E9Y4
WARNING: Physical disk problems have been found.
------------------------
Adapter Information
------------------------
Adapters present: 2
Adapter 0:
[Product Name] PERC H730P Mini
[Firmware Version] 4.250.00-4402
[Serial Number] 59N05XI
[Memory] 2048MB
[Alarm Device] Absent
[Alarm Enabled] Disabled
[Memory Correctable Errors] 0
[Memory Uncorrectable Errors] 0
[Battery Backup Unit] Present
[Restore HotSpare on Insertion] Disabled
[Use Full Disk Encryption Only] Yes
[Security Keys Assigned] No
[Security Keys Failed] No
[Security Keys Not Backed Up] No
[Virtual Drives] 2
[VD's Degraded] 0
[VD's Offline] 0
[Failed Disks] 0
Adapter 1:
[Product Name] PERC H830 Adapter
[Firmware Version] 4.250.00-4402
[Serial Number] 59N001Bpvscan
[Memory] 2048MB
[Alarm Device] Absent
[Alarm Enabled] Disabled
[Memory Correctable Errors] 0
[Memory Uncorrectable Errors] 0
[Battery Backup Unit] Present
[Restore HotSpare on Insertion] Disabled
[Use Full Disk Encryption Only] Yes
[Security Keys Assigned] No
[Security Keys Failed] No
[Security Keys Not Backed Up] No
[Virtual Drives] 4
[VD's Degraded] 0
[VD's Offline] 0
[Failed Disks] 0
--------------------------------------------------------------------
At this point I attempted to “MANUALLY tear down the DAC with the by doing the following;
Stopped nwappliance and nwconcentrator service(s)
Un-mounted the /var/netwitness/concentrator filesystems
Ran the lvscan, vgscan, and pvscan commands and ran the appropriate lvremove, vgremove, and pvremove commands.
Edited the /etc/fstab to comment out entries that the script has made for the DAC.
***Be sure to comment out any entries made in /etc/fstab for the DAC. If you do not, and run he below command and reboot before re-running the script your system will not boot! ***
Lastly I ran the following command to remove all the drives for the 830 adapter.
#/opt/MegaRAID/MegaCli/MegaCli64 -CfgLdDel –Lall –a(x)
It was at this it was discovered that the script was failing because of a conflict associated with the < /dev/sdb > physical volume. We came up with the following workaround to “trick” the script into running properly. See in RED below…
--------------------------------------------------------------------
root@sanitizedhostname arrayCfg]# fdisk -l /dev/sdb
Disk /dev/sdb: 1999.8 GB, 1999844147200 bytes
255 heads, 63 sectors/track, 243133 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00090505
Device Boot Start End Blocks Id System
[root@sanitizedhostname arrayCfg]# pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created
[root@sanitizedhostname arrayCfg]# vgcreate dummy /dev/sdb
Volume group "dummy" successfully created
[root@root@sanitizedhostname arrayCfg]# lvcreate -l +100%FREE dummy -n dummy
Logical volume "dummy" created
[root@sanitizedhostname arrayCfg]# pvscan
PV /dev/sdd VG index lvm2 [928.72 GiB / 0 free]
PV /dev/sdc VG concentrator lvm2 [25.47 TiB / 0 free]
PV /dev/sdb VG dummy lvm2 [1.82 TiB / 0 free]
PV /dev/sda2 VG VolGroup00 lvm2 [255.97 GiB / 87.97 GiB free]
Total: 4 [28.44 TiB] / in use: 4 [28.44 TiB] / in no VG: 0 [0 ]
--------------------------------------------------------------------
At this point we were able to start over “SUCESSFULLY” running the 2.2 script with first the “- - action init” and then the “- - action add” options.
--------------------------------------------------------------------
[root@sanitizedhostname arrayCfg]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-root
20G 1.7G 17G 10% /
tmpfs 64G 0 64G 0% /dev/shm
/dev/sda1 256M 61M 182M 25% /boot
/dev/mapper/VolGroup00-usrhome
4.0G 136M 3.7G 4% /home
/dev/mapper/VolGroup00-opt
20G 53M 20G 1% /opt
/dev/mapper/VolGroup00-rsaroot
10G 36M 10G 1% /opt/rsa
/dev/mapper/VolGroup00-tmp
20G 33M 20G 1% /tmp
/dev/mapper/VolGroup00-var
20G 91M 20G 1% /var
/dev/mapper/VolGroup00-rabmq
20G 35M 20G 1% /var/lib/rabbitmq
/dev/mapper/VolGroup00-varlog
16G 60M 16G 1% /var/log
/dev/mapper/VolGroup00-nwhome
30G 90M 30G 1% /var/netwitness
/dev/mapper/VolGroup00-vartmp
4.0G 33M 4.0G 1% /var/tmp
/dev/mapper/concentrator-root
30G 33M 30G 1% /var/netwitness/concentrator
/dev/mapper/index-index
929G 34M 929G 1% /var/netwitness/concentrator/index
/dev/mapper/concentrator-sessiondb
2.6T 33M 2.6T 1% /var/netwitness/concentrator/sessiondb
/dev/mapper/concentrator-metadb
23T 33M 23T 1% /var/netwitness/concentrator/metadb
/dev/mapper/index0-index
929G 34M 929G 1% /var/netwitness/concentrator/index0
/dev/mapper/concentrator0-sessiondb
2.6T 33M 2.6T 1% /var/netwitness/concentrator/sessiondb0
/dev/mapper/concentrator0-metadb
23T 33M 23T 1% /var/netwitness/concentrator/metadb0
/dev/mapper/index1-index
929G 34M 929G 1% /var/netwitness/concentrator/index1
/dev/mapper/concentrator1-sessiondb
2.6T 33M 2.6T 1% /var/netwitness/concentrator/sessiondb1
/dev/mapper/concentrator1-metadb
23T 33M 23T 1% /var/netwitness/concentrator/metadb1
/dev/mapper/index2-index
929G 34M 929G 1% /var/netwitness/concentrator/index2
/dev/mapper/concentrator2-sessiondb
2.6T 33M 2.6T 1% /var/netwitness/concentrator/sessiondb2
/dev/mapper/concentrator2-metadb
23T 33M 23T 1% /var/netwitness/concentrator/metadb2
--------------------------------------------------------------------
After this we cleaned up the /dev/sdb entries were made…
[root@sanitizedhostname arrayCfg]# lvchange -an dummy
[root@sanitizedhostname arrayCfg]# lvremove dummy
Logical volume "dummy" successfully removed
[root@sanitizedhostname arrayCfg]# vgremove dummy
Volume group "dummy" successfully removed
[root@sanitizedhostname arrayCfg]# pvremove /dev/sdb
Labels on physical volume "/dev/sdb" successfully wiped
RE-Start nwappliance and nwconcentrator service(s)
2016-11-29 08:55 AM
Just ran into this yesterday. Thanks for the write up GKKxwgBy9sU0nCbYviD6G5r7VXv0d4jsoi0NgriKbn4=!