1 ACCEPTED SOLUTION
Accepted Solutions
Okay this is version 1. I'm sure that there are going to be issues but lets give it a go.
In the script you define a set of meta keys that you want to use in your feed.
METAKEYS="os.name os.platform os.sname os.type os.ver brand model bot.info clientinfo.type clientinfo.name clientinfo.sname clientinfo.ver clientinfo.plat"
You also define a link field. This is the field that you will matching on in your feed. In my case I made it ip.src but you could chnage it to what you want the index field of your feed to be. So in your original request this will be user.dst
LINKFIELD="ip.src"
Other Parameters are
MAXRESULTS=200
USER=admin
PASSWORD=netwitness
BROKER=192.168.123.249:50103
OUTPUTFEEDFILE=/var/www/html/myfeed.csv
What the script does is as follows:
- Find LINKFIELD values where at least one of the desired metakeys exists
- Takes the FIRST value found and puts this in a feed.
- Creates a feed in the file name specified in the variable OUTPUTFEEDFILE.
Limitations:
- I needed to install dos2unix as the output from the CURL command contained DOS ^M end of line characters and I needed to replace them with Unix \n
- As mentioned, if the LINKFIELD matches mutliple values then only the first is considered. In this example if I was using ip.src and then a request was made from a two different browsers, then the results for the first request are captured.
- The feed is refreshed every hour. This is controlled by line $PreviousTime
- Run the file as a cron job to create the feed.
Remember to recreate a recurrent feed in the SA GUI to make use of your feed.
I've attached the script but here it is. I'm sure it can be improved!
rawurlencode() {
local string="${1}"
local strlen=${#string}
local encoded=""
local pos c o
for (( pos=0 ; pos<strlen ; pos++ )); do
c=${string:$pos:1}
case "$c" in
[-_=.:/\&\?~a-zA-Z0-9] ) o="${c}" ;;
* ) printf -v o '%%%02x' "'$c"
esac
encoded+="${o}"
done
echo "${encoded}" # You can either set a return variable (FASTER)
REPLY="${encoded}" #+or echo the result (EASIER)... or both...
}
CurrentTime=$(date -u +"%Y-%b-%d %H:%M:%S")
#echo "Current Time is: " $CurrentTime
PreviousTime=$(date -u -d '1 hour ago' +"%Y-%b-%d %H:%M:%S")
#echo "1 Hour Ago was: " $PreviousTime
# The Callback key we will use in our feed
LINKFIELD="ip.src"
#The Meta Keys that we want in our feed
METAKEYS="os.name os.platform os.sname os.type os.ver brand model bot.info clientinfo.type clientinfo.name clientinfo.sname clientinfo.ver clientinfo.plat"
EXISTS=$(echo "$METAKEYS" |sed 's/ / exists ||/g')
SELECT=$(echo "$METAKEYS" |sed 's/ /,/g')
EXISTS="$EXISTS exists"
#echo "EXISTS: " $EXISTS
#echo "SELECT: " $SELECT
MAXRESULTS=200
USER=admin
PASSWORD=netwitness
BROKER=192.168.123.249:50103
OUTPUTFEEDFILE=/var/www/html/myfeed.csv
# First Get All Link Fields where our Meta Keys Exist
#URL="http://$BROKER/sdk?msg=query&size=$MAXRESULTS&query=select distinct($LINKFIELD) where $EXISTS&time='$PreviousTime'-'$CurrentTime'&force-content-type=text/plain"
#echo "URL: " $URL
URL=$( rawurlencode "$URL")
#echo "Encoded URL: " $URL
curl -s --user "$USER:$PASSWORD" "$URL" |grep value |cut -d " " -f 9 |cut -d "=" -f 2 >/tmp/linkfields.txt
#curl -s --user "$USER:$PASSWORD" "$URL" >/tmp/linkfields.output
# Print out the header for our file
echo "#"$LINKFIELD,$SELECT > $OUTPUTFEEDFILE
while read link; do
# We know that our linkfield has values for out METAKEYS so lets get the values over the last hour
URL="http://$BROKER/sdk?msg=query&size=$MAXRESULTS&query=select $SELECT where $LINKFIELD=$link&time='$PreviousTime'-'$CurrentTime'&force-content-type=text/plain"
URL=$( rawurlencode "$URL")
# Our Result will contain ^M Line endings so we replace these with newline characters
$(curl -s --user "$USER:$PASSWORD" "$URL" >/tmp/result.txt)
dos2unix -q -o /tmp/result.txt
OUTPUT=$link
OUTPUT+="^"
for meta in $METAKEYS
do
#cat /tmp/result.txt |grep value |grep -m 1 $meta >/tmp/$link.$meta.output.txt
OUTPUT+=$(cat /tmp/result.txt |grep value |grep -m 1 $meta |cut -d "=" -f 6 |sed 's/ type//' |xargs )
OUTPUT+="^"
done
echo $OUTPUT >> $OUTPUTFEEDFILE
done </tmp/linkfields.txt
