User Agent to Device/OS/Application

[AlexeyFedorov]

Hello,

Somobody has solution how I can transformate data from user.agent metakey (Log and Packet) to some like Device, OS, Application via Feed or LUA Parser? I found some FlexParser in RSA Live, but it is not I want. I found some site like Complete List of iOS User-Agent Strings | Enterprise iOS with DB of iPhone/iPad/IPod, but data is old and I can't create suitable Feed.

I found interesting progect GitHub - piwik/device-detector: The Universal Device Detection library will parse any User Agent and detect the browser,… and his web version Device Detector Demo​

It is what I want, but how transformate it in LUA Parser or Feed?

[AlexeyFedorov]

Maybe you are right, but I think it is not unique solution. I hope find solution situable for all customer...

[AlexeyFedorov]

I guess I found solution situable for all customer. We can use Application Rule and regex. In the source code of Device Detector has strings like:

  regex: '(?:CPU OS|iPh(?:one)?[ _]OS|iOS)[ _/](\d+(?:[_\.]\d+)*)'

  name: 'iOS'

  version: '$1'

I prepare set of Application Rule for most common OS, Device and Application and share for all. I need few days...

[DavidWaugh1]

Hi that will be a lot of work. You will basically be duplicating the work of the DeviceDetector project.

So far I have:

- Written a PHP Web Page that I can submit my own User Agents to

- Written a Small Script that will submit the Useragent String to the WebPage and output it in feed friendly manner.

As an example:

./UserAgentInfo.sh "Mozilla/5.0 (phone; CPU iPhone OS 9_2_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13D15 Safari/601.1"

returns:

Mozilla/5.0 (phone; CPU iPhone OS 9_2_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13D15 Safari/601.1^browser^Mobile Safari^MF^9.0^^iOS^IOS^9.2.1^^AP^iPhone

My feed seprater will be ^

The column Names are:

Useragent String^Client type^Client Name^ Client Short Name^Client Version^Client Platfrom^OS Name^OS Short Name^OS Version^OS Platform^Brand^Model

My next plan is to schedule a job to download all the uncategorised Client Usernames from a broker and then feed each line into the script to create a feed file.

I'll keep you posted!

[AlexeyFedorov]

This is not dedup work of DeviceDetector project. I copy-past from their format to format application rule. Now I have finished move OS (oss.yml file from project) to Application Rule and attach result of my work. I ran into a problem with version number and not correct detect Windows NT and GNU/Linux OS. I can't put version number into rule name for some rules. I aslo can't stop filtered Windows NT if user agent string contains both Windows X and Windows NT values.

Maybe you devide you work and will go by modules? On the first stage move oss.yml to LUA parser (conver or include) and then go to browsers and etc.? It will be very cool, you can make many very good LUA parsers situable for any customers. At this time in RSA Live not present this content.

I also need LUA parser that can replace "+" to space on the user.agent metakey for correct work with user agent string from MS IIS.

[DavidWaugh1 · Accepted solution]

Okay I have a working solution.

- I created a PHP Page page on a Webserver based on the DeviceDetector project (Install the DeviceDetector project on your own PhP enabled webserver then use copy the file hello.php to the same directory where DeviceDetector.php resides)

- I created a shellscript that sits on my PhP Enabled webserver that queries my broker for UserAgents on Service 80 and then creates a feed based on the values returned.

-Edit the line in the UserAgentInfo.sh to replace user 'admin:netwitness' 'http://192.168.123.249:50103' with an account and broker that is able to get values. I used admin netwitness because I was a bit lazy and because this is just a test system!

The script creates a file called /var/www/html/useragentfeed.csv that can then be read by Security Analytics as a feed. Note in the feed definition file make sure that the callback key you are using is case-insensitive. The attached DeviceDetector.xml file contains a feed definition example.

The solution could be tweaked for your own environment but as a proof of concept I'm happy with it 😃

Here is a Demo of the hello.php webpage. It basically takes the useragent in the Query and then outputs the findings.

The final output will be the Feed file which I have attached.

I added the following meta keys to my concentrators to use the feed:

<!--DeviceDetector Feed -->

<key description="ClientInfo Type" format="Text" level="IndexValues" name="clientinfo.type" defaultAction="Open" valueMax="500000"/>

<key description="ClientInfo Name" format="Text" level="IndexValues" name="clientinfo.name" defaultAction="Open" valueMax="500000"/>

<key description="ClientInfo Short Name" format="Text" level="IndexValues" name="clientinfo.sname" defaultAction="Open" valueMax="500000"/>

<key description="ClientInfo Version" format="Text" level="IndexValues" name="clientinfo.ver" defaultAction="Open" valueMax="500000"/>

<key description="ClientInfo Platform" format="Text" level="IndexValues" name="clientinfo.plat" defaultAction="Open" valueMax="500000"/>

<key description="OS Type" format="Text" level="IndexValues" name="os.type" defaultAction="Open" valueMax="500000"/>

<key description="OS Name" format="Text" level="IndexValues" name="os.name" defaultAction="Open" valueMax="500000"/>

<key description="OS Short Name" format="Text" level="IndexValues" name="os.sname" defaultAction="Open" valueMax="500000"/>

<key description="OS Version" format="Text" level="IndexValues" name="os.ver" defaultAction="Open" valueMax="500000"/>

<key description="OS Platform" format="Text" level="IndexValues" name="os.platform" defaultAction="Open" valueMax="500000"/>

<key description="Brand" format="Text" level="IndexValues" name="brand" defaultAction="Open" valueMax="500000"/>

<key description="Model" format="Text" level="IndexValues" name="model" defaultAction="Open" valueMax="500000"/>

[AlexeyFedorov]

I will check you solution.

Can I ask you about LUA Parser what will replace all "+" to " " (space) on metakey? From MS IIS user agent string come like "Mozilla/5.0+(Windows+NT+6.2;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/47.0.2526.106+Safari/537.36". I need transformate it to "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" for correct determinate.

And second question. I found yaml to CSV online converter YAML To CSV Converter . If put to this convertor file oss.yml and convert it - you get result like:

"Tizen[ /]?(\d+[\.\d]+)?","Tizen","$1"

"Sailfish|Jolla","Sailfish OS",""

"(?:Ali)?YunOS[ /]?(\d+[\.\d]+)?","YunOS","$1"

"Windows Phone (?:OS)?[ ]?(\d+[\.\d]+)","Windows Phone","$1"

"XBLWP7|Windows Phone","Windows Phone",""

First field - regex, Second Filed - OS name, Third field - OS version

Can you wrap this array to LUA perser and merge (options) second and third fields? This parser will very simple support in future if you can check regex from first filed to metakey and write result (merged second and third fileds) to another metakey(s), for example OS (version). What do you think about it? This structura can be unique for some other parsers.

[DavidWaugh1]

I'll take a look at the first question. 😃

[AlexeyFedorov]

David, can I put DeviceDetector project on some RSA Security Analytics component? For example on SA Server? I don't have any other Web Servers...

[DavidWaugh1]

Here is the LUA Parser that will replace + in IIS User Agents with Spaces.

[DavidWaugh1]

Hi Alexey I wouldnt recommend it and there isnt a web server on the SA Server than runs a PHP webserver either as far as I know.

The alternative would be to run the PHP script from the command line but I didnt have much luck with that. My PHP skills are a little limited!