Feed Limitations -- large data sets

[JoeGumke]

Have any customers ingested large feeds (feeds with several thousand + items)?

I'm curious what kind of decoder performance degredation we will see with large feeds processing our data. I'm sure we have to deal with calculating EPS and feed size .

My use case is to ingest vulnerability data with 20k + assets, so wondering what other customers impact have been with something similar. We have a need to identify / associate known vulnerabilities to tune in/out certain vulns from alerts.

[EricPartington]

I have tried the Cisco Umbrella DNS top 1Million and had timeout issues using the UI.  Splitting it up into smaller chunks has worked.  Also keep in mind the number of columns of data that comes along with the number of rows will impact as well.  A vuln list with 10 columns will be larger than a vuln list with 2 columns.

I don't have a hard number for rows and columns before timeout which seems to be the limiting factor in what will work or not.  Once on the decoder I believe its compiled and has minimal impact on decoder performance.  THe written meta as a result of a match could potentially have more impact on system performance (average meta per session might increase, which could affect aggregation rates and retention period for concentrators and archivers).

[JoshuaWaterloo]

I have a customer that is leveraging CRITs to generate feeds for NetWitness and here is the stats from them:

How big/how many lines are the feeds passed to NetWitness?  The largest is roughly 165k lines with 12 fields per line which translates to a  82 MB CSV file.  

How many feeds are produced in CRITs for NetWitness?  5 in total.

How often are they run?  Once an hour.

How long does it take to process?  Less than 2 seconds.  Loaded binary feed CRITSAlias (v0), load time: 1.896 seconds.

A colleague of mine has generated feeds with up to 500,000 lines and the CSV was pulled, compiled, and pushed within a couple minutes in their customer’s environment.  I know that isn't hard and fast numbers but I agree with Eric.  Best option would be to test your feeds and if there are performance issues split into smaller chucks.