2016-07-29 11:47 AM
One of my existing customer is using an AIO appliance for logs with 32 TB DAC, so now there is a change in their RSA SA architecture, now they are going to re-image the AIO appliance to Log Hybrid appliance, so just want to know & understand few points related to this change in the architecture.
I have few questions on the above scenario.
1- Concentrator - Like as the AIO have Concentrator running on it - so is it required to take backup of indexed data which would lying under Concentrator partition. If i take backup of it will i able to restore the same on the Log Hybrid after the re-image.
And by which process i need to take the backup of Concentrator's indexed data, because (as such there is no proper procedure is available on sadocs for AIO data backup)
2- Log Decoder - As same like for concentrator, do i need to take the data backup of raw logs from the AIO box and how i take the data backup from the AIO appliance (as such there is no proper procedure is available on sadocs for AIO data backup) and will i able to restore & use the same data in the Log Hybrid server after backup.
3- DAC - Customer have 32 TB DAC attached to the same AIO appliance, so just want to know what best possible we can do with the DAC, is it required to first clear/remove (backup) all the data from the DAC before I re-mount the same with Log Hybrid.
If i take the backup of the DAC data too then how to restore on DAC (because there is no guide i am to search to do data backup & restore of data from the running DAC)
And will i able to re-mount the same with Log Hybrid and will i able to view restored data on DAC from the Log Hybrid server after the backup & restore procedure.
It would be great help if you can add some points which i have missed or else please add your valuable points if there is any best practices to do in these kind of scenario.
Please advise on the above points. Thank you.
Regards,
Deepanshu Sood.
2016-07-30 12:28 AM
Do you have a requirement to save all of the current data? I've done rebuilds of this sort, but I did a clean sweep. I just did a backup of all custom content the customer had deployed and wiped all old data. After the reinstall, I added in all of the customers custom content.
2016-08-01 09:02 AM
Consider simply removing the services from the appliance you no longer intend to run. You can do this with the "rpm -e <package>" command.
You might need to modify the mongo database to on the AIO but since you aren't running the jettysrv service that might not be essential.
You would also have to strip out the certificates on the AIO appliance and generate new certificates before you added the appliance into the new SA Server.
You would also need to be careful about changing the name and IP address of the appliance but there are KB articles that explain how to do this.
You might lose a little space doing this but it would be minimal and you would not need to backup 11TB of data off the AIO appliance.
Hope that helps.
2016-08-02 11:07 AM
Here is what I have done in the past (when doing CentOS 5 to CenOS6 rebuilds on hybrids, so converting an AIO to hybrid will work pretty much the same) is to roll enough data off the system to make room for all the data on the DAC and move all the internal data to the DAC.
- If space is needed, use timeroll to reduce the data set size (determine how far back your data goes by looking at the "stats" for the database & index in explore view:
From explore view of each service, right click on "database" folder and select properties:
Concentrator databases
Select "timeroll" from dropdown menu, for parameters: type=session,meta days=180 (would remove any session and meta data older than 180 days adjust as necessary for your needs)
Decoder databases
Select "timeroll" from dropdown menu, for parameters: type=session,meta,packet days=60 (would remove any session and meta data older than 60 days adjust as necessary for your needs)
Index (both)
From SSH session to device trim the index slices to a date just beyond the date of the timeroll
cd /var/netwitness/[concentrator|decoder]/index.
Each "managed-values=NNNNN" is an index slice (you may need to look at the files inside the slice folder to determine the actual date of the files, ins index repairs can change the folder dates)
Example: To remove managed-values-1000 - 1500, use:
for x in {1000..1500..1};do rm -rf managed-values-$x;done
You can determine the space being used by each DB by doing:
du -sh /var/netwitness/ [decoder|concentrator]/[index|sessiondb|metadb|packetdb]
Once there is enough space on the DAC to move the internal data:
Stop collection and aggregation, remove aggregation link from concentrator -> decoder.
Stop all services (nwdecoder, nwconcentrator, nwappliance)
Create a temp location on the DAC for the internal filesystem structure and copy all the data over to the DAC.
Also backup the configuration to the DAC or another safe location:
Backup the following configuration files for appliance, concentrator and decoder and external DAC mount points.
/etc/netwitness/ng/NwAppliance.cfg
/etc/netwitness/ng/NwConcentrator.cfg
/etc/netwitness/ng/NwDecoder.cfg
/etc/netwitness/ng/index-concentrator-custom.xml
/etc/netwitness/ng/index-decoder-custom.xml
/etc/fstab (you only need the lines for the external storage array from this file, not the whole file)
Reboot and re-image the AIO to a Hybrid, configure IP/Hostname info (use same ones as old system
Do not enter SA IP address yet, hit <ctrl> C, when asked.)
Manually edit /etc/yum.repos.d/RSASoftware.repo file on newly imaged device, to point to the SA server repo:
For a 10.5 SA server:
[RSASoftware]
name=Base
baseurl=http://<SA server IP>/rsa/updates/
enabled=1
gpgcheck=0
sslverify=1
For a 10.6 SA server:
[RSASoftware]
name= $sarelease - Base
baseurl=http://puppetmaster.local/rsa/updates/$sarelease/
enabled=1
gpgcheck=0
sslverify=1
Upgrade new box to same level as old box following normal update procedures.
Restore Configuration & data:
Stop nwdecoder, nwconcentrator, nwappliance services.
Restore backup files, (do NOT overwrite fstab, just copy the lines from the old fstab file for the external DAC mount points and paste them at the end of the fstab on the new box)
Attempt to mount the external file systems (mount -a) if errors, check lvscan to see if LV's are active, if not activate them (see attached doc), make sure they mount clean.
Copy old internal data back to the proper internal filesystems. Remove backup data from DAC.
Log out, log in, enter SA head IP address, and Enable the box in SA as normal.
Once enabled, reboot
System should come up operational.
May need to remove ad re-add concentrator to broker/reporting engine/ESA/etc. Since it will have new puppet keys/certs.