Custom Feed - Domains not indexed in Investigation

[EvanRamos1]

Howdy all.

Just recently created two custom feeds. One is IPs , the other attempt is unique domain names going into custom meta. The domain names are not being indexed, and no errors are seen on my log decoder.  For the most part, i followed this: https://community.rsa.com/docs/DOC-78049  , everything is lowercase being imported.

Any ideas what might be wrong? Or setup incorrectly?

My feed setup is as follows -

Type: Non-IP

Index Column: 1

Service Type: 0

Truncate Domain: Enabled

Callback Key(s): domain.dst (i've tried alias.host, domain too).

XML -

<FlatFileFeed comment="#" separator="," path="otxdomainiocs.csv" name="otxdomainiocs">
<MetaCallback truncdomain="true" apptype="0" valuetype="Text" name="domain.dst"/>
<LanguageKeys><LanguageKey valuetype="Text" name="otxc2.domainioc"/>
</LanguageKeys><Fields><Field type="index" index="1"/>
<Field key="otxc2.domainioc" type="value" index="2"/>

Example data being imported and wanting to index. 

anh.phimhainhat.net,otx_c2domainioc
dalat.dalat.dulichovietnam.net,otx_c2domainioc
dalat.dulichovietnam.net,otx_c2domainioc
dalat.hanoi.dulichovietnam.net,otx_c2domainioc
danang.dalat.dulichovietnam.net,otx_c2domainioc

[david_waugh]

In you metacallback line you need ignorecase="true"

Sent from my iPhone

[EvanRamos1]

Thanks David. I tired that and still no go. Hmmmmm....I'll keep messing with values in the xml i guess.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<FDF>
  <FlatFileFeed comment="#" separator="," path="otxdomainiocs.csv" name="otxdomainiocs">
    <MetaCallback truncdomain="true" apptype="0" valuetype="Text" ignorecase="true" name="domain.dst" />
    <LanguageKeys>
      <LanguageKey valuetype="Text" name="otxc2.domainioc"/>
    </LanguageKeys>
    <Fields>
      <Field type="index" index="1"/>
      <Field key="otxc2.domainioc" type="value" index="2"/>
    </Fields>
  </FlatFileFeed>
</FDF>‍‍‍‍‍‍‍‍‍‍‍‍‍

[BrianKeenan]

Since this is new meta you are creating have you added the otxc2.domainioc to your concentrator custom index file and set it to index values?

[EvanRamos1]

Yup. Here is what i put in the index files -

index-concentrator-custom.xml -

<key description="otxc2-domainioc" format="Text" level="IndexValues" name="otxc2.domainioc" valueMax="250000" />‍

index-logdecoder-custom.xml -

<key description="otxc2-domainioc" format="Text" level="IndexNone" name="otxc2.domainioc" />‍

index-decoder-custom.xml -

<key description="otxc2-domainioc" format="Text" level="IndexNone" name="otxc2.domainioc" />‍

[SravanKoneti1]

Hi Evan,

Please try below xml contents.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<FDF>
<FlatFileFeed comment="#" separator="," path="otxdomainiocs.csv" name="otxdomainiocs">
<MetaCallback valuetype="Text" ignorecase="true" name="domain.dst" />
<LanguageKeys>
<LanguageKey valuetype="Text" name="otxc2.domainioc"/>
</LanguageKeys>
<Fields>
<Field type="index" index="1" key="domain.dst"/>
<Field type="value" index="2" key="otxc2.domainioc"/>
</Fields>
</FlatFileFeed>
</FDF>

[EricPartington]

can you post the link if possible to the original feed data from  OTX? i'd like to see if there is a better way to import this that does not require a manual XML and custom metakey to  make your life a bit easier.

[EvanRamos1]

Hey Eric,

Sorry for a late reply. Yeah, i can give a snippet of the original feed data from OTX (pre adjusting data to NW feed).  Since this is public API stuff, no big deal.    https://file.io/45PGV6    <-just a .zip file , no tricks. lol. 

So i did test the feed by putting in a domain (that is indexed already in domain metakey) , and it worked. So i'm guessing this is setup right i just dont have any of the bad domain IOCs in my network.