NetWitness Community

RSAAdmin · ‎2013-08-28

Hi All,

I have been having some serious frustrations trying to populate the email.src meta field.

I have asked RSA directly how to populate this and they informed me that its used for Logs not packets. however the email parser that comes with LIVE content only populates 'email' making it almost impossible to drill based on a sender. (for instance only wanting outbound mail instead of everything an email address sent or received.

Anyhow, I was told I would need a custom parser to achieve my objective, and this is what I have come up with:

<parser Name="From Mail Parser" desc="Version 0.1">
<!-- Parser Created by David McClennon @ Integralis.com -->
<declaration>
<!-- Declare meta keys to be used in this parser-->
<meta format="Text" key="email.dst" name="recipient"/>
<meta format="Text" key="email.src" name="sender"/>
 
 
<!-- Mail from and To identifiers to Match -->
<token name="MailFrom" value="MAIL FROM:<" options="linestart"/>
 
 
<!-- Position Identifiers -->
<number name="vMailPosition" scope="stream" />
<number name="vMailStop" scope="steam" />
<!-- Variables for holding strings -->
 <string name="vEmailAddress" scope="stream" />
</declaration>
 
<!-- Evaluate From mail match -->
 <match name="MailFrom">
 <!-- Find ">" within 512 bytes -->
 <find name="vMailPosition" value="&#x3C;" length="512">
 <!-- when found, assign contents between current potistion and vmailPosition to vEmailAddress -->
 <read name="vEmailAddress" length="$vMailPosition">
 <register name="sender" value="$vEmailAddress" />
 </read>
 </find>
 </match>         
    </match>
 
</parser>

But for love nor money can I get it to work properly. I have tried a number of match combinations including encoding the "MAIL FROM:" in various formats, but cannot seem to get it to match or register the email address in the email.src register.

If anyone can see what I have done wrong, or have any suggestions, I would be most grateful.

Regards

Dave

RSAAdmin · ‎2013-08-28

A couple of tips- make sure you have a meta key for email.src in your local investigator index. For live deployment you also have to have these keys on the decoders/concentrators/brokers.

I presume you are working with a pcap of mail traffic for testing. After you create your parser and drop it into your parsers directory, you have to restart investigator and open it back up. This essentially reloads the parsers just like it was on a decoder. Now check the logs of Investigator under Help-> show log. Search the logs for this parser name. If it's broken, it usually drops a hint as to why. If it loads it will be listed along with every other local parser that loaded.

Just glancing at your parser, you reference vmailstop- this is either unneeded or should be a token, but its not referenced again. If you parser needs to locate "<" then that should be a token, registered above as the vmailstop. Then your find statement should be simply find vmailstop.

View solution in original post

RSAAdmin · ‎2013-08-28

A couple of tips- make sure you have a meta key for email.src in your local investigator index. For live deployment you also have to have these keys on the decoders/concentrators/brokers.

I presume you are working with a pcap of mail traffic for testing. After you create your parser and drop it into your parsers directory, you have to restart investigator and open it back up. This essentially reloads the parsers just like it was on a decoder. Now check the logs of Investigator under Help-> show log. Search the logs for this parser name. If it's broken, it usually drops a hint as to why. If it loads it will be listed along with every other local parser that loaded.

Just glancing at your parser, you reference vmailstop- this is either unneeded or should be a token, but its not referenced again. If you parser needs to locate "<" then that should be a token, registered above as the vmailstop. Then your find statement should be simply find vmailstop.

RSAAdmin · ‎2013-08-28

<?xml version="1.0" encoding="utf-8"?>

</declaration>

<register name="meta"

value="$vMailFrom" />

</read>

</find>

</match>

</parser>

</parsers>

RSAAdmin · ‎2013-08-28

I think the above will get you closer. Now you can create a separate parser for the recipient as well.

RSAAdmin · ‎2013-08-28

I found this lying around in my filesystem. No idea who made it or if it works. Looks like it requires a bunch of keys. Your mileage may vary, no complaints accepted if this doesn't work or if it blows up your decoder.

<?xml version="1.0" encoding="utf-8"?>

<!--

****

META DECLARATIONS

****

-->

<!--

****

STRINGS

****

-->

<!--

****

NUMBERS

****

-->

<!--

****

TOKENS

****

-->

</declaration>

<!--

****

FILTER MATCH

****

-->

</match>

</match>

<!--

****

REMAINING MATCHES

****

-->

<!--<match name="myToken_Pop_subject">

<end/>

</if>

</if>

</read>

</find>

</match>-->

<end/>

</if>

<find name="myNum_offset" value=">" length="64">

</if>

</read>

</find>

</match>

<end/>

</if>

<find name="myNum_offset" value=">" length="64">

</if>

</read>

</find>

</match>

<end/>

</if>

<find name="myNum_offset" value=">" length="64">

</if>

</read>

</find>

</match>

<end/>

</if>

<find name="myNum_offset" value=">" length="64">

</if>

</read>

</find>

</match>

<end/>

</if>

</if>

</read>

</find>

</match>

<end/>

</if>

</if>

</read>

</find>

</match>

<!-- <match name="myToken_Attachment">

<end/>

</if>

</if>

</read>

</find>

</match> -->

<end/>

</if>

</if>

</read>

</find>

</match>

<end/>

</if>

</if>

</read>

</find>

</match>

<!-- Commenting out here for performance reasons

<end/>

</if>

</if>

</read>

</find>

</match>

<end/>

</if>

<end/>

</if>

</if>

</read>

</find>

</match>

-->

<end/>

</if>

</if>

</read>

</find>

</if>

</read>

</find>

</move>

</find>

</match>

<end/>

</if>

<end/>

</if>

</match>

</parser>

</parsers>

RSAAdmin · ‎2013-08-28

That is awesome thank you. (ps. I want your filesystem )

RSAAdmin · ‎2013-08-28

For that parser to work all I need to do is register the meta values on "decoder/concentrator/broker" and add the parser to the Flex.parser correct? Im not missing anything am I?

Regards

Dave

RSAAdmin · ‎2013-08-28

Here would be the keys you would need. Paste them into the concentrator/broker but:

Change the level to IndexNone on the decoders or you will meet your doom!

RSAAdmin · ‎2013-08-28

Fielder wrote:

Here would be the keys you would need. Paste them into the concentrator/broker but:
Change the level to IndexNone on the decoders or you will meet your doom!

Learn't that the hard way some time ago thank you for the confirmation. That is almost word for word what I had put in the index

RSAAdmin · ‎2013-08-28

1 final quick question if I may. how would I add meta to identity if the from is an internal domain?

I have a meta value "soc.misc" that I would like to give a value of "Internal" which I could then use in the following context

custom drill: "soc.misc !="Internal" which would in turn leave me with all inbound mail only.

I am guessing something like this:

<meta name="misc" key="soc.misc" format="Text" />
 
</declaration>
<!--snip-->
<match name="myToken_FROM">
      <if name="myNum_FoundEHLO" notequal="1">
        <end/>
      </if>
      <find name="myNum_offset" value="&#x3E;" length="64">
        <read name="myString_Temp" length="$myNum_offset">
          <if name="myString_Temp" notequal="$myString_Empty">
            <register name="fromaddy" value="$myString_Temp" />
<!-- Check for internal domain name -->          
  <find length="64" name="myNum_offset" value="mydomain.com"/>
          <!-- if found register misc key value-->
               <register name="misc" value="Internal"/>
               </find>
              <assign name="myString_Temp" value="$myString_Empty"/>
          </if>
        </read>
      </find>
    </match>

Do I need to move the pointer before running the find?

Thanks again for your help.

Dave

NetWitness Community

Mail From email.src Custom Parser