The Limits of Feeds !!

[OmarGarciaGilio]

Hello
I have a couple of doubts about Feeds (custom ones):

1st: How can I make that the feed work only for a specific device(s) type(s). For example: I want to use as a callback key user.dst meta to populate 3 more metas, but as this user info is valid only for a specific aplication, I would like that only matchs the user.dst for that specific device type.

2nd: The csv or text file that upload, How bigger can be? Right now, I Have my csv file for my initial upload, and It have 1.13GB (more than 14 millions of lines). Do you think that the system can handle it?

thanks in advance.

[Anonymous]

That does seem to be a large feed, but I think the decoders can still handle it.  However, you might be able to use a multi-indexed feed to achieve your goal of user.dst AND a specific device.type feed.  The link below may give you what you need in order to develop it.

000035599 - Creating custom feeds with multiple indexed meta keys for RSA Security Analytics 10.6.x 

Chris

[OmarGarciaGilio]

Thanks for the info. So far I got this xml "DataCliAmdocs.xml":

<?xml version="1.0" encoding="utf-8"?>
<FDF xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="feed-definitions.xsd">
<FlatFileFeed name="DataCliAmdocs"
path="DataCliAmdocs.csv"
separator="|"
comment="#">

<MetaCallback name="DeviceType">
<Meta name="device.type" valuetype="Text" ignorecase="true"/>
</MetaCallback>

<MetaCallback name="Instance">
<Meta name="instance" valuetype="Text" ignorecase="true"/>
</MetaCallback>

<LanguageKeys>
<LanguageKey name="tdp.dni.cliente" valuetype="Text"/>
<LanguageKey name="middlename" valuetype="Text"/>
<LanguageKey name="tdp.data.cliente" valuetype="Text"/>
</LanguageKeys>

<Fields>
<Field index="1" type="index" key="DeviceType"/>
<Field index="2" type="index" key="Instance"/>
<Field index="3" type="value" key="tdp.dni.cliente"/>
<Field index="4" type="value" key="middlename"/>
<Field index="5" type="value" key="tdp.data.cliente"/>
</Fields>
</FlatFileFeed>
</FDF>

And this file "DataCliAmdocs.csv" (Just a sample, the real have over 14M):

#amdocs|CUSTOMER_KEY|ID|FULL_NAME|VAR
amdocs|99999742|N/A|PICA PIEDRA,PEDRO PANCHO|R:Resistor--1524:Amarillo
amdocs|99999757|N/A|SALAS LIMON,FRIENDLY|R:Resistor--1524:Amarillo
amdocs|99999772|99999999|CAVER NARIO,EVE|R:Resistor--1524:Amarillo
amdocs|99999784|N/A|CARDENAS CADENAS,MARI|R:Resistor--1524:Amarillo
amdocs|99999848|88888888|GENE JUNE,USER|R:Resistor--1530:Verde
amdocs|99999860|N/A|GER LUA,FRIENDLY|R:Resistor--1524:Amarillo

Well I upload (with WinScp) both file to "LogDec" on directory "/root" then I run "NwConsole -c feed create DataCliAmdocs" but I got thi error:

[root@DSC-SA-DEC ~]# NwConsole -c feed create DataCliAmdocs
RSA Security Analytics Console 10.6.2.1
Copyright 2001-2017, RSA Security Inc. All Rights Reserved.
>feed create DataCliAmdocs
File not found.

Then I try use the wizard (just 1 MetaCallback = instance, and delete the 1st column from csv and choose | as separator), but after show me the yellow progress bar (final step) turn red and say 'failed'. Also try loading the xml on wizard (Adv options) but got "xml error" as message.

I don't have a clue what I'm doing wrong @_@

I'm thinking that actually there is a limit for size of a feed.

[BrianKeenan]

Have you tried with a smaller csv just for testing purposes?

Also you might have better luck using NwConsole to compile the feed.

https://community.rsa.com/docs/DOC-46349 

Hope that helps.

[OmarGarciaGilio]

Hi Brian,

Thanks for the answer, I already tried to compile the feed with using "NwConsole" but got an error and the feed file never be generate. Instead I tried to split my file in lower pieces and then generate my feed with one of them. Finally I notice that the bigger file that it can handle has 62213 lines (4.45MB aprox), if I put just one more line in the file then get the error again.
Well the bad news is that as my file is 14 million of lines so that would made more than 200 files for my feed. That mean that the limit of lines that a feed can handle is 62213 (v10.6.2).

[JohnKisner]

Since you want it to be for a specific device type and 1.13GB is way too large for the feed creation process you may want to create a feed for each device type. That would make each feed a logical separator though creating over 200 feeds is pretty daunting. Can you tell us exactly the end result you are looking for as there may be a better way to accomplish what you are looking for as an end result.

[OmarGarciaGilio]

Hi John

Basically I need to enrich Client data (full name and ID), the goal is make a rule that show all the registers where client ID (get from feed, I thought) is equal to user ID (this meta already exist). But as there are more than 14 millions of clients, the feed is not the way.

[JohnKisner]

Omar,

Are you trying to connect Usernames with IP address and workstation names? If that is what you are attempting to do you can use the Identity feed option if you are collecting from an Active Directory server. Please let me know if this is actually what you are attempting to do and if so I can provide some instructions for that.