Automated Threat Detection for Log Decoder

[NikolayKlender]

New component of ESA server Automated Threat Detection (ATD) is very promising. According to manual it analyzes HTTP packets which are parsed by the Decoder and sent to the ESA device. But if you have Log Decoder only you are out of luck (according to manual). When I asked support team of RSA about possibility to use ATD with logs from proxy device (squid, BlueCoat, etc) I received following answer:

Automated Threat Detection is an analytics engine that examines your HTTP data hence only supported for packets in combination with LUA parser.

There are no plans to expand it with logs functionality.

http://sadocs.emc.com/0_en-us/088_SA106/50_Alrt/66_C2Thrt/00_TDInfo

But all this seems strange because web proxy logs contains all necessary information.

If we look at meta from http_lua parser and contains of file /opt/rsa/esa/topology/C2.topology we can understood that following meta are analyzed by ATD:

• action (GET, POST, etc)
• ip_src
• alias_host - web domain
• client - user agent
• referer
• time
• service (must be equal 80)

Log decoder do not fill service meta, that is why ATD do not analyze logs from LogDecoder. Thats why we need to modify parser to extract required meta and inject service meta of value 80. Lets take cacheflowelff parser as an example.

1) In  table-map-custom.xml we associate new xservice field with service meta

          <mapping envisionName="xservice" nwName="service" flags="None" format="UInt32"/>

2) open v20_cacheflowelffmsg.xml and edit content section of GET, POST, etc message id.

So here we inject xservice field with value 80 (will be moved to service meta), extract User-Agent to agent field (will be moved to client meta), and extract referer field. Please look table-map.xml for more information about mapping fields from envision parsers to netwitness metas

content="&lt;@xservice:80&gt;&lt;@msg:*PARMVAL($MSG)&gt;&lt;@fld63:*PARMVAL(username)&gt;&lt;@fld61:*PARMVAL(h_code)&gt;&lt;@bytes:*CALC(sbytes,+,rbytes)&gt;&lt;@domain:*URL($DOMAIN,url)&gt;&lt;@web_root:*URL($ROOT,url)&gt; &lt;@event_time:*EVNTTIME($MSG,'%W-%G-%F %N:%U:%O',fld1,fld2)&gt;&lt;@event_source:*PARMVAL(hostname1)&gt;&lt;@dhost:*PARMVAL(web_host)&gt;date=&lt;fld1&gt;,time=&lt;fld2&gt;,time-taken=&lt;processing_time&gt;,c-ip=&lt;saddr&gt;,s-action=&lt;action&gt;,sc-status=&lt;resultcode&gt;,sc-bytes=&lt;sbytes&gt;,cs-bytes=&lt;rbytes&gt;,cs-method=&lt;web_method&gt;,cs-user=&lt;username1&gt;,cs-username=&lt;username&gt;,cs-uri-username=&lt;username&gt;,s-hierarchy=&lt;h_code&gt;,cs-host=&lt;web_host&gt;,rs(Content-Type)=&lt;content_type&gt;,cs-uri-port=&lt;dport&gt;,s-ip=&lt;hostip&gt;,r-ip=&lt;dtransaddr&gt;,r-supplier-ip=&lt;fld37&gt;,r-dns=&lt;fld38&gt;,c-port=&lt;sport&gt;,cs-category=&lt;category&gt;,cs-uri-scheme=&lt;network_service&gt;,duration=&lt;duration&gt;,s-supplier-ip=&lt;daddr&gt;,cs-auth-group=&lt;group_object&gt;,s-supplier-name=&lt;fld68&gt;,sc-filter-result=&lt;disposition&gt;,sc-filter-category=&lt;filter&gt;,cs(User-Agent)=&lt;agent&gt;,x-virus-id=&lt;virusname&gt;,s-sitename=&lt;service&gt;,cs-uri=&lt;url&gt;,cs-uri-path=&lt;webpage&gt;,x-exception-id=&lt;fld88&gt;,cs-categories=&lt;category&gt;,cs(Referer)=&lt;referer&gt;,cs-uri-query=&lt;web_query&gt;,cs-uri-extension=&lt;web_extension&gt;,cs(Cookie)=&lt;web_cookie&gt;,s-computername=&lt;hostname1&gt;,s-port=&lt;network_port&gt;,cs-version=&lt;version&gt;,cs-auth-groups=&lt;group_object&gt;,cs-uri-stem=&lt;fld3&gt;,localtime=&lt;fld5&gt;,x-bluecoat-application-name=&lt;fld4&gt;,x-bluecoat-application-operation=&lt;fld6&gt;,s-icap-status=&lt;fld7&gt;,s-icap-info=&lt;fld8&gt;"

3)restart logdecoder

stop nwlogdecoder && start nwlogdecoder

4) configure ATD according to manual. After 24 hours you should see some new incidents

[MarkKarlstrand]

Hi Nikolay,

Thank you for posting this article to the community. It’s great to have passionate SA users, such as yourself experimenting with new ways to extract value out of the product. As you point out, the meta fields used by the detection model may be parsed from web proxy log events. This is actually by design as RSA does intend to release a similar model that can operate on web proxy logs. The current model however has only been certified to operate against HTTP packet data. This means that running the current model against log sources may give unexpected results compared to the same traffic captured as PCAPs. For example, the packet HTTP data is sessionized by TCP where web proxy data is made up of individual request/response transactions. This difference may require tweaks to the model in order to deliver similar detection results. So for now we do not recommend that customers attempt to utilize the model against log data in production but we would love to work with yourself and any other customers that have a lab environment in which to try out this and other experiments with SA.

Thanks,

Mark Karlstrand

mark.karlstrand@rsa.com