Prepare Virtual or Cloud Storage
This section describes how to set up virtual or cloud storage for the following types of component hosts:
- Decoder, Log Decoder, Concentrator, Archiver
- NW Server, ESA Primary, ESA Secondary and Malware Analysis
- Log Collector
- Endpoint Log Hybrid
- Additional Endpoint Log Hybrid Partitions
- UEBA
Decoder, Log Decoder, Concentrator, ArchiverDecoder, Log Decoder, Concentrator, Archiver
Virtual or Cloud NetWitness hosts for Decoders, Log Decoders, Concentrators, and Archivers need block storage attached. Make sure that the allocated storage meets all of the storage requirements. Specifically, make sure that the required storage volumes are created (see "Required NetWitness Platform Storage Volumes" in Storage Requirements), and:
- At least two Block Devices are created for Decoders (meta /session and packet volumes)
- At least two block devices are created for Concentrators (index and meta volumes)
- Ensure that block devices can meet the minimum IOPS for expected ingestion rates
Attach the allocated storage to the NetWitness host by following the hosting platforms native procedure.
- VmWare – Vsphere Console (add disk to VM)
- Hyper-V – Manager Console (add disk to VM)
- Azure – Add Managed Disks to virtual instance
- AWS – Add EBS Storage to virtual instance
-
Google Cloud Platform (GCP) - Add storage to virtual instance
After the storage is attached to the virtual host, proceed to "Task 3 - Allocate Block Devices to Partitions, Volume Groups, and Logical Volumes" in Configure Storage Using the REST API.
NW Server, ESA Primary, ESA Secondary and Malware AnalysisNW Server, ESA Primary, ESA Secondary and Malware Analysis
For an extension of /var/netwitness/ partition, attach an external volume.
Run lsblk to get the physical volume name and run the following commands:
1. pvcreate <pv_name> (for example, pv_name is /dev/sdc)
2. vgextend netwitness_vg00 /dev/sdc
3. lvresize --resizefs --extents +100%FREE /dev/netwitness_vg00/nwhome
4. xfs_growfs /dev/netwitness_vg00/nwhome
NetWitness recommends the following partition definitions. However, you can change these values based on the retention days.
LVM | Folder | Block Storage |
---|---|---|
/dev/netwitness_vg00/nwhome | /var/netwitness/ | Refer to the Cloud Provider Block Storage setup (storage) tables. |
Log CollectorLog Collector
For an extension of /var/netwitness/ partition, attach an external volume
Run lsblk to get the physical volume name and run the following commands:
1. pvcreate <pv_name> (for example, pv_name is dev/sdc)
2. vgextend netwitness_vg00 /dev/sdc
3. lvresize --resizefs --extents +100%FREE /dev/netwitness_vg00/nwhome
4. xfs_growfs /dev/netwitness_vg00/nwhome
NetWitness recommends the following partition definitions. However, you can change these values based on the retention days.
LVM | Folder | Block Storage |
---|---|---|
/dev/netwitness_vg00/nwhome | /var/netwitness/ | Refer to the Cloud Provider Block Storage setup (storage) tables. |
Log Decoder
Virtual Drive Space Ratios
The following table provides optimal configurations for packet and log hosts.
Log Decoder | |||
Persistent Datastores | Cache Datastore | ||
PacketDB | SessionDB | Meta DB | Index |
100% as calculated by Sizing & Scoping Calculator | 1 GB per 1000 EPS of traffic sustained provides 8 hours cache | 20 GB per 1000 EPS of traffic sustained provides 8 hours cache | 0.5 GB per 1000 EPS of traffic sustained provides 4 hours cache |
Extending File Systems
Follow the below instructions to extend the file systems.
Attach an external disk for extension of /var/netwitness/ partition, create an external disk with suffix as nwhome, attach other external disks for LogDecoder database partition. For extending /var/netwitness partition, follow these steps:
No other partition should reside on this volume, only to be used for /var/netwitness/
1. Ensure you have added a new disk. For more information, see Task 1. Add New Disk.
2. Execute lsblk and get the physical volume name, suppose if you had add attach one 2TB disk.
3. pvcreate <pv_name> suppose the PV name is /dev/sdc.
4. vgextend netwitness_vg00 /dev/sdc.
5. lvextend –L 1.9T /dev/netwitness_vg00/nwhome.
or,
lvextend -l +100%FREE /dev/netwitness_vg00/nwhome.
6. xfs_growfs /dev/mapper/netwitness_vg00-nwhome.
Other partitions are also required. Create the following partitions on the logdecodersmall volume group.
Folder |
LVM |
Volume Group |
---|---|---|
/var/netwitness/logdecoder | decoroot | logdecodersmall |
/var/netwitness/logdecoder/index | index | logdecodersmall |
/var/netwitness/logdecoder/metadb |
metadb |
logdecodersmall |
/var/netwitness/logdecoder/sessiondb | sessiondb | logdecodersmall |
Follow these steps to create the partitions mentioned in the table above:
1. Execute lsblk and get the physical volume names from the output.
2. pvcreate /dev/sdd.
3. vgcreate –s 32 logdecodersmall /dev/sdd.
4. lvcreate –L <disk_size> -n <lvm_name> logdecodersmall.
5. mkfs.xfs /dev/logdecodersmall/<lvm_name>.
6. Repeat steps 4 and 5 for all the LVM’s mentioned.
The following partition should be on volume group LogDecoder.
Folder |
LVM |
Volume Group |
---|---|---|
/var/netwitness/logdecoder/packetdb | packetdb | logdecoder |
Follow these steps:
1. Execute lsblk and get the physical volume names from the output.
2. pvcreate /dev/sde.
3. vgcreate –s 32 logdecoder /dev/sde.
4. lvcreate –L <disk_size> -n packetdb logdecoder.
5. mkfs.xfs /dev/logdecoder/packetdb.
NetWitness recommends below sizing partition for LogDecoder (Can be changed based on the retention days).
LVM |
Folder |
Size |
Disk Type |
---|---|---|---|
/dev/netwitness_vg00/nwhome | /var/netwitness/ | 1TB | HDD |
/dev/logdecodersmall/decoroot | /var/netwitness/logdecoder | 10GB | HDD |
/dev/logdecodersmall/index |
/var/netwitness/logdecoder/index |
30GB |
HDD |
/dev/logdecodersmall/metadb | /var/netwitness/logdecoder/metadb | 3TB | HDD |
/dev/logdecodersmall/sessiondb |
/var/netwitness/logdecoder/sessiondb |
370GB |
HDD |
/dev/logdecoder/packetdb | /var/netwitness/logdecoder/packetdb | 18TB | HDD |
Create each directory and mount the LVM on it in a serial manner, except /var/netwitness which will be already created.
Create the folder /var/netwitness/logdecoder and mount on /dev/logdecodersmall/decoroot then create the other folders and mount them.
After that add the below entries in /etc/fstab in the same order and mount them using mount –a.
/dev/logdecodersmall/decoroot /var/netwitness/logdecoder xfs noatime,nosuid 1 2
/dev/logdecodersmall/index /var/netwitness/logdecoder/index xfs noatime,nosuid 1 2
/dev/logdecodersmall/metadb /var/netwitness/logdecoder/metadb xfs noatime,nosuid 1 2
/dev/logdecodersmall/sessiondb /var/netwitness/logdecoder/sessiondb xfs noatime,nosuid 1 2
/dev/logdecoder/packetdb /var/netwitness/logdecoder/packetdb xfs noatime,nosuid 1 2
Concentrator
Virtual Drive Space Ratios
The following table provides optimal configurations for packet and log hosts.
Concentrator | ||
Persistent Datastores | Cache Datastores | |
Meta DB | SessionDB Index | Index |
Calculated as 10% of the PacketDB required for a 1:1 retention ratio | 30 GB per 1TB of PacketDB for standard multi protocol network deployments as seen at typical internet gateways | 5% of the calculated MetaDB on the Concentrator. Preferred High Speed Spindles or SSD for fast access |
Log Concentrator | ||
Persistent Datastores | Cache Datastores | |
Meta DB | SessionDB Index | Index |
Calculated as 100% of the PacketDB required for a 1:1 retention ratio | 3 GB per 1000 EPS of sustained traffic per day of retention | 5% of the calculated MetaDB on the Concentrator. Preferred High Speed Spindles or SSD for fast access |
Extending File Systems
Attach external disk for extension of /var/netwitness/ partition, Create an external disk with suffix as nwhome, attach other external disks for Concentrator database partition.
For extending /var/netwitness partition follow below steps:
No other partition should reside on this volume, only to be used for /var/netwitness/.
1. Ensure you have added a new disk. For more information, see Task 1. Add New Disk.
2. Execute lsblk and get the physical volume name, for example if you attach one 2TB disk.
3. pvcreate /dev/sdc suppose the PV name is /dev/sdc.
4. vgextend netwitness_vg00 /dev/sdc.
5. lvextend –L 1.9T /dev/netwitness_vg00/nwhome.
or,
lvextend -l +100%FREE /dev/netwitness_vg00/nwhome.
6. xfs_growfs /dev/mapper/netwitness_vg00-nwhome.
The following partitions are also required on volume group concentrator.
Folder |
LVM |
Volume Group |
---|---|---|
/var/netwitness/concentrator | root | concentrator |
/var/netwitness/concentrator/sessiondb | sessiondb | concentrator |
/var/netwitness/concentrator/metadb |
metadb |
concentrator |
Follow these steps:
1. Execute lsblk and get the physical volume names from the output.
2. pvcreate /dev/sdd.
3. vgcreate –s 32 concentrator /dev/sdd.
4. lvcreate –L <disk_size> -n <lvm_name> concentrator.
5. mkfs.xfs /dev/concentrator/<lvm_name>.
6. Repeat steps 4 and 5 for all the LVM’s mentioned.
Below partition should be on volume group index.
Folder |
LVM |
Volume Group |
---|---|---|
/var/netwitness/concentrator/index | index | index |
Follow these steps:
1. Execute lsblk and get the physical volume names from the output.
2. pvcreate /dev/sde.
3. vgcreate –s 32 index /dev/sde.
4. lvcreate –L <disk_size> -n index index.
5. mkfs.xfs /dev/index/index.
NetWitness recommends below sizing partition for Concentrator (Can be changed based on the retention days).
LVM |
Folder |
Size |
Disk Type |
---|---|---|---|
/dev/netwitness_vg00/nwhome | /var/netwitness/ | 1TB | HDD |
/dev/concentrator/root | /var/netwitness/concentrator | 10GB | HDD |
/dev/concentrator/metadb |
/var/netwitness/concentrator/metadb |
3TB |
HDD |
/dev/concentrator/sessiondb | /var/netwitness/concentrator/sessiondb | 370GB | HDD |
/dev/index/index |
/var/netwitness/concentrator/index |
2TB |
HDD |
Create each directory and mount the LVM on it in a serial manner, except /var/netwitness which will be already created.
Create the folder /var/netwitness/concentrator and mount on /dev/concentrator/root then create the other folders and mount them.
After that add the below entries in /etc/fstab in the same order.
/dev/concentrator/root /var/netwitness/concentrator xfs noatime,nosuid 1 2
/dev/concentrator/sessiondb /var/netwitness/concentrator/sessiondb xfs noatime,nosuid 1 2
/dev/concentrator/metadb /var/netwitness/concentrator/metadb xfs noatime,nosuid 1 2 2
/dev/index/index /var/netwitness/concentrator/index xfs noatime,nosuid 1 2
Archiver
The following partition is required for the Archiver volume group.
Folder |
LVM |
Volume Group |
---|---|---|
/var/netwitness/archiver | archiver | archiver |
Follow these steps:
1. Execute lsblk and get the physical volume names from the output.
2. pvcreate /dev/sde.
3. vgcreate –s 32 archiver /dev/sde.
4. lvcreate –L <disk_size> -n archiver archiver.
5. mkfs.xfs /dev/archiver/archiver.
Attach an external disk for extension of /var/netwitness/ partition, create an external disk with suffix as nwhome, attach other external disks for Archiver database partition.
For extending /var/netwitness partition follow these steps:
No other partition should reside on this volume, only to be used for /var/netwitness.
1. Ensure you have added a new disk. For more information, see Task 1. Add New Disk.
2. Execute lsblk and get the physical volume name, suppose if you had add attach one 2TB disk.
3. pvcreate /dev/sdc suppose the PV name is /dev/sdc.
4. vgextend netwitness_vg00 /dev/sdc.
5. lvextend –L 1.9T /dev/netwitness_vg00/nwhome.
or,
lvextend -l +100%FREE /dev/netwitness_vg00/nwhome.
6. xfs_growfs /dev/mapper/netwitness_vg00-nwhome.
NetWitness recommends the following sizing partition for the Archiver (Can be changed based on the retention days).
LVM |
Folder |
Size |
Disk Type |
---|---|---|---|
/dev/netwitness_vg00/nwhome | /var/netwitness/ | 1TB | HDD |
/dev/archiver/archiver | /var/netwitness/archiver | 4TB | HDD |
Create each directory and mount the LVM on it in a serial manner, except /var/netwitness which will be already created.
After that add the below entries in /etc/fstab in the same order.
/dev/archiver/archiver /var/netwitness/archiver xfs noatime,nosuid 1 2
Decoder
Virtual Drive Space Ratios
The following table provides optimal configurations for packet and log hosts.
Decoder | |||
Persistent Datastores | Cache Datstore | ||
PacketDB | SessionDB | Meta DB | Index |
100% as calculated by Sizing & Scoping Calculator | 6 GB per 100Mb/s of traffic sustained provides 4 hours cache | 60 GB per 100Mb/s of traffic sustained provides 4 hours cache | 3 GB per 100Mb/s of traffic sustained provides 4 hours cache |
Extending File Systems
Attach an external disk for extension of /var/netwitness/ partition, create an external disk with suffix as nwhome, attach other external disks for decoder database partition. For extending /var/netwitness partition follow these steps:
No other partition should reside on /var/netwitness/.
1. Ensure you have added a new disk. For more information, see Task 1. Add New Disk.
2. Execute lsblk and get the physical volume name, suppose if you had add attach one 2TB disk.
3. pvcreate /dev/sdc.
4. vgextend netwitness_vg00 /dev/sdc.
5. lvextend –L 1.9T /dev/netwitness_vg00/nwhome.
or,
lvextend -l +100%FREE /dev/netwitness_vg00/nwhome.
6. xfs_growfs /dev/mapper/netwitness_vg00-nwhome.
The following four partitions should be on the decodersmall volume group.
Folder |
LVM |
Volume Group |
---|---|---|
/var/netwitness/decoder | decoroot | decodersmall |
/var/netwitness/decoder/index | index | decodersmall |
/var/netwitness/decoder/metadb |
metadb |
decodersmall |
/var/netwitness/decoder/sessiondb | sessiondb | decodersmall |
Follow these steps:
1. Execute lsblk and get the physical volume names from the output.
2. pvcreate /dev/sdd.
3. vgcreate –s 32 decodersmall /dev/sdd.
4. lvcreate –L <disk_size> -n <lvm_name> decodersmall.
5. mkfs.xfs /dev/decodersmall/<lvm_name>.
6. Repeat steps 4 and 5 for all the LVM’s mentioned
The following partition should be on the decoder volume group.
Folder |
LVM |
Volume Group |
---|---|---|
/var/netwitness/decoder/packetdb | packetdb | decoder |
1. Execute lsblk and get the physical volume names from the output.
2. pvcreate /dev/sde.
3. vgcreate –s 32 decoder /dev/sde.
4. lvcreate –L <disk_size> -n packetdb decoder.
5. mkfs.xfs /dev/decoder/packetdb.
NetWitness recommends the following sizing partition for the Decoder (Can be changed based on the retention days).
LVM |
Folder |
Size |
Disk Type |
---|---|---|---|
/dev/netwitness_vg00/nwhome | /var/netwitness/ | 1TB | HDD |
/dev/decodersmall/decoroot | /var/netwitness/decoder | 10GB | HDD |
/dev/decodersmall/index |
/var/netwitness/decoder/index |
30GB |
HDD |
/dev/decodersmall/metadb | /var/netwitness/decoder/metadb | 3TB | HDD |
/dev/decodersmall/sessiondb |
/var/netwitness/decoder/sessiondb |
370GB |
HDD |
/dev/decoder/packetdb | /var/netwitness/decoder/packetdb | 18TB | HDD |
Create each directory and mount the LVM on it in serial manner, except /var/netwitness which will be already created.
Create the folder /var/netwitness/decoder and mount on /dev/decodersmall/decoroot then create the other folders and mount them.
After that add the below entries in /etc/fstab in the same order and mount them using mount –a.
/dev/decodersmall/decoroot /var/netwitness/decoder xfs noatime,nosuid 1 2
/dev/decodersmall/index /var/netwitness/decoder/index xfs noatime,nosuid 1 2
/dev/decodersmall/metadb /var/netwitness/decoder/metadb xfs noatime,nosuid 1 2
/dev/decodersmall/sessiondb /var/netwitness/decoder/sessiondb xfs noatime,nosuid 1 2
/dev/decoder/packetdb /var/netwitness/decoder/packetdb xfs noatime,nosuid 1 2
Endpoint Log Hybrid
The total disk size required depends on the data retention period. You can use the below per day disk usage indicative values to calculate the required disk size for your deployment. For example, to retain 30 days of data, multiply the below per day disk usage values with 30.
The following table provides disk usage for one full scan. The full scan disk usage values are based on the below event count:
-
Files count -1100
-
Processes count -100
- Dlls count - 500
-
Drivers count -150
-
Services count - 500
-
Tasks count -100
Endpoint Log Hybrid(50K Advance Agents - Disk usage per full scan) | |||||
---|---|---|---|---|---|
MetaDB | PacketDB | SessionDB | Index |
Total |
|
Log Decoder |
220 GB | 12 GB | 5 GB |
NA |
237 GB |
Concentrator | 230 GB | NA | 5 GB | 6 GB |
241 GB |
MongoDB | NA | NA | NA | NA |
35 GB (First full scan) 30 GB (Subsequent per scan increase) |
The following tables provide per day disk usage for tracking data. The total tracking events per agent per day is 29000.
Endpoint Log Hybrid (50K Advance Agents - Tracking data without Expanded Network Visibility) | |||||
---|---|---|---|---|---|
MetaDB | PacketDB | SessionDB | Index |
Total |
|
Log Decoder |
1500 GB | 140 GB | 46 GB |
NA |
1,686 GB |
Concentrator | 1600 GB | NA | 46 GB | 30 GB |
1,676 GB |
MongoDB | NA | NA | NA | NA |
35 GB (First full scan) 1.5 GB (Tracking data per day increase) |
The following tables provide per day disk usage for tracking data. Total tracking events per agent per day is 33000
Endpoint Log Hybrid (50K Advance Agents - Tracking data with Expanded Network Visibility) | |||||
---|---|---|---|---|---|
MetaDB | PacketDB | SessionDB | Index |
Total |
|
Log Decoder |
1800 GB | 152 GB | 55 GB |
NA |
2007 GB |
Concentrator | 1900 GB | NA | 55 GB | 36 GB |
1991 GB |
MongoDB | NA | NA | NA | NA |
35 GB (First full scan) 1.5 GB (Tracking data per day increase) |
The following table provides per day disk usage for insight agents. The total tracking data per agent per day is 10800 plus 1 full scan daily.
Endpoint Log Hybrid (50K Insights Agents with Expanded Network Visibility) | |||||
---|---|---|---|---|---|
MetaDB | PacketDB | SessionDB | Index |
Total |
|
Log Decoder |
500 GB | 52 GB | 18 GB |
NA |
570 GB |
Concentrator | 600 GB | NA | 18 GB | 13 GB |
631 GB |
MongoDB | NA | NA | NA | NA |
35 GB (First full scan) 30 GB (Subsequent per scan increase) |
The following table provides Endpoint Agents sizing based on the feature.
Feature | Description | Agent or Endpoint Server | ||||
---|---|---|---|---|---|---|
Endpoint Only |
Only scan and tracking data |
Maximum 50K Endpoint Agents only | ||||
Windows Logs Only | Only Windows Logs from agents. Assuming 20K events per second supported by Hybrid. |
|
||||
File Collection Only | Only File Collection from agents. Assuming 20K events per second supported by Hybrid |
|
||||
Endpoint and Windows Logs |
|
Maximum 15K (approximately) Agents:
Plus
|
||||
Endpoint, Windows Logs and File Collection |
|
Maximum 10K (approximately) Agents:
Plus
Plus
|
Extending File Systems
For Endpoint Server, attach external disk for extension of /var/netwitness/ partition, create an external disk with suffix as nwhome.
Follow these steps:
- Ensure you have added a new disk. For more information, see "Task 1. Add New Disk" in the Virtual Hosts Installation Guide for NetWitness Platform. Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.
2. Execute lsblk and get the physical volume name
3. pvcreate <pv_name> suppose the PV name is /dev/sdc
4. vgextend netwitness_vg00 /dev/sdc
5. lvresize --resizefs --extents +100%FREE /dev/netwitness_vg00/nwhome
6. xfs_growfs /dev/mapper/netwitness_vg00-nwhome
NetWitness recommended partition for Endpoint Server (can be changed based on the retention days).
LVM | Folder | Size | Disk Type |
---|---|---|---|
/dev/netwitness_vg00/nwhome | /var/netwitness/ | 6TB | HDD |
For Mongo DB, attach external disk for extension of /var/netwitness/mongo partition, create an external disk with suffix as nwhome.
Follow these steps:
-
Ensure you have added a new disk. For more information, see "Task 1. Add New Disk" in the Virtual Hosts Installation Guide. Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.
2. Execute lsblk and get the physical volume name
3. pvcreate <pv_name> suppose the PV name is /dev/sdc1
4. vgextend hybrid /dev/sdc1
5. lvresize --resizefs --extents +100%FREE /dev/hybrid-vlmng
6. xfs_growfs /dev/mapper/hybrid-vlmng
NetWitness recommended partition for Mongo DB (Can be changed based on the retention days). Minimum recommended size for var/netwitness is 500 GB.
LVM | Folder | Size | Disk Type |
---|---|---|---|
/dev/hybrid-vlmng | /var/netwitness/mongo | 6TB | HDD |
Additional Endpoint Log Hybrid PartitionsAdditional Endpoint Log Hybrid Partitions
The following partition should be on the volume group endpoint and should be in a single RAID 0 array.
Folder | LVM | Volume Group |
---|---|---|
/var/netwitness/mongo | hybrid-mongo | endpoint |
/var/netwitness/concentrator | concentrator-concroot |
endpoint |
/var/netwitness/concentrator/index |
hybrid-concinde |
endpoint |
/var/netwitness/logdecoder | hybrid-ldecroot | endpoint |
Run lsblk to get the physical volume name and run the following commands:
- pvcreate /dev/md0
- vgcreate –s 32 endpoint /dev/md0
- lvcreate –L <disk_size> -n <lvm_name> endpoint
- mkfs.xfs /dev/ endpoint /<lvm_name>
- Repeat the above steps for all the LVMs mentioned.
NetWitness recommends the following partitions. However, you can change these values based on the retention days.
LVM | Folder | Block Storage |
---|---|---|
/dev/netwitness_vg00/nwhome | /var/netwitness/ | Refer to the Cloud Provider Block Storage setup (storage) tables. |
/dev/endpoint/hybridmongo |
/var/netwitness/mongo | Refer to the Cloud Provider Block Storage setup (storage) tables. |
/dev/endpoint/concentratorconcroot | /var/netwitness/concentrator | Refer to the Cloud Provider Block Storage setup (storage) tables. |
/dev/endpoint/hybridconcinde | /var/netwitness/concentrator/index | Refer to the Cloud Provider Block Storage setup (storage) tables. |
/dev/endpoint/hybridldecroot | /var/netwitness/logdecoder | Refer to the Cloud Provider Block Storage setup (storage) tables. |
UEBAUEBA
The following procedure attaches an external disk and extends the /var/netwitness/ partition. You must use nwhome as the eternal disk suffix. This procedure illustrates how to add a 2TB disk.
Note: /var/netwitness is the only partition that can reside on this volume.
- List the physical volume name.
lsblk (for example, dev/mapper/sdc) - Extend the /var/netwitness/ partition.
pvcreate <pv_name>where pv_name is dev/mapper/sdc
vgextend netwitness_vg00 /dev/mapper/sdc
lvresize --resizefs --extents +100%FREE /dev/mapper/netwitness_vg00/nwhome
xfs_growfs /dev/mapper/netwitness_vg00-nwhome
This partition is the NetWitness recommended partition for UEBA. You can change it based on retention days.