This website uses cookies. By clicking Accept, you consent to the use of cookies. Click Here to learn more about how we use cookies.
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
NetWitness Discussions
- NetWitness Community
- Discussions
- Re: Python Script to Email Report on Concentrator Indexes
-
Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Python Script to Email Report on Concentrator Indexes
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2016-12-28 12:30 PM
I improved an existing Python script that RSA Professional Services provided me a while back. TnCNqFbwCBwNT2aUaQOZPvE3tSL3jJyeOvABeYMBjJM=
I am pasting here for your reference. It is useful to run against your concentrators from say the SA head unit to collect all the indexes max meta values and email you a report.
It has built@ in help via --help and requires modification before use in your environment.
1. Update lines 18 and 19 to set the sender and receiver email addresses.
2. Update line 27 so it points to a flat file (one per line) of either the IP address, hostname or alias of your concentrators.
3. Update line 28 if you'd like to set the appropriate location for your log file to be written out to.
4. Update line 29, 30 to set the service account credentials used to authenticate. Yes it is in plaintext on this server, no I don't have an alternative unless RSA's API will accept tokens and offer a way to generate them.
5. Update line 233 to set your internal SMTP server or set an open relay if you use one. This script doesn't support SMTP authentication to send mail. Assumes open internal relays.
Two ways to run it.
1. Individual concentrator as such:
index_profile.py --host sa-concentrator1
2. A newline/carriage return delimited file of IP addresses/hostnames:
index_profile.py --file
If you want to improve the logging level of the script you can update line 49:
handler.setLevel(logging.INFO)
Update the INFO part to DEBUG or WARN for example.
Once I've verified on an individual host it works, I setup in crontab to run every 4 hours. I check the emails daily and see if any meta keys pop out. Keep in mind unless you know exactly how frequently your indexes roll, it can be hard to determine how frequently to run the script.
#!/usr/bin/env python
# Author: kddiens
# Changelog:
# 03/29/2016 - kdd - created off sample code from RSA, added email function and formatting
# 03/30/2016 - kdd - restructing into a loop for main so we can go across multiple hosts
# 03/31/2016 - kdd - fixes and cleanup, adding script location to html output
# 12/21/2016 - kdd - Adding internal script logging to ease updating, writing, etc
###########################################################################################
import requests, csv, sys, argparse, json, smtplib, time, datetime, os, logging
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
########################
### Global Variables ###
########################
sender = 'sample@domain.com'
receiver = 'sample@domain.com'
today = datetime.datetime.today()
now = datetime.datetime.now()
t1 = now.strftime("%H:%M:%S")
t2 = now.strftime("%H")
d1 = today.strftime("%Y-%m-%d")
body = []
hosts = []
filename = os.path.join("/root/scripts/applianceList/concentratorlist")
log_file = '/root/scripts/check-indexes/index_profile_concentrators' + str(d1) + '_' + str(t2) + '.log'
user = 'serviceaccountname'
password = 'SECRET PASSWORD'
port = "50105"
language_payload = { 'msg' : 'language', 'force-content-type' : 'text/plain', 'expiry' : '600' }
inspect_payload = { 'msg' : 'inspect', 'force-content-type' : 'application/json', 'expiry' : '600' }
###############
### Logging ###
###############
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
#set debug logging
#logging.basicConfig(level=logging.DEBUG)
#logger.setLevel(logging.DEBUG)
# create file handler for logger
handler = logging.FileHandler(log_file)
handler.setLevel(logging.INFO)
# create logging format
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
# add the handles to the logger
logger.addHandler(handler)
###############
# MAIN #
###############
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--host", help="Host IP or Name", dest="host", action="store")
parser.add_argument("--file", help="Input File", dest="fflag", action="store_true")
args = parser.parse_args()
if not args.host and not args.fflag:
logger.info('Hostname not specified and file flag not set, unable to continue...')
sys.exit("Hostname not specified with --host and input file flag not defined via --file")
if args.fflag and args.host:
logger.info('Hostname and file flag not null, these are exclusive arguments, unable to continue..')
sys.exit("--file and --host parameters are exclusive, they cannot be combined, try again")
if args.host:
logger.info('Hostname exists, passing param to function for individual host %s...' % (args.host))
body.append("<br>The following metakeys on <b>%s</b> are over used and require index config custom updates: <br>" % (args.host))
#collect_meta(args.host,args.ssl) potential to add SSL flag during runtime to toggle if HTTP or HTTPS is used
collect_meta(args.host)
elif args.fflag:
logger.info('File flag set, using file declared in global variable %s...' % (filename))
logger.info('Changing state to build_hosts() function.')
build_hosts()
filtered_hosts = filter(None, hosts)
for host in filtered_hosts:
body.append("<br>The following metakeys on <b>%s</b> are over used and require index config custom updates: <br>" % (host))
#collect_meta(host,args.ssl)
collect_meta(host)
email(body)
############################################################
## Read in concentrator list line by line and create list ##
############################################################
def build_hosts():
logger.info('Attempting to open file %s to build hosts', filename)
f = open(filename, 'r')
for line in f.read().split('\n'):
logger.info('Adding host %s from file into hosts array', line)
hosts.append(line)
f.close()
###########################################
## Function to collect the meta per host ##
###########################################
def collect_meta(host):
logger.info('Host passed to function is %s', host)
# Construct the URI, will add path later
host_uri = "http://" + host + ":" + port
logger.info('Host URI constructed is %s', host_uri)
# Do a try catch and get a HTTP response from the system
try:
index_language = requests.get(host_uri + '/index',
verify=False,
auth=(user, password),
params=language_payload,
timeout=5)
logger.info('Full URL to locate index langauge is %s', index_language.url)
if index_language.status_code == 200:
logger.info('w00t! HTTP 200 response!')
elif index_language.status_code != 200:
logger.warn('Program received status code %s for host %s using URL %s', index_language.status_code, host, index_language.url)
body.append("WARNING: Cannot reach URL: %s on host %s <br>" % (index_language.url,host))
except requests.exceptions.Timeout as e:
logger.warn('Network timeout trying to reach host %s, this may be due to firewall restrictions or an issue with the service we are trying to contact.', host)
body.append("WARNING: Host %s is unreachable! <br>" % (host))
return
try:
index_inspect = requests.get(host_uri + '/index',
verify=False,
auth=(user, password),
params=inspect_payload,
timeout=5)
logger.info('Full URL to locate index inspect page is %s', index_inspect.url)
if index_inspect.status_code == 200:
logger.info('w00t! HTTP 200 response!')
elif index_inspect.status_code != 200:
logger.warn('Program received status code %s for host %s using URL %s', index_inspect.status_code, host, index_inspect.url)
body.append("WARNING: Cannot reach URL: %s on host %s <br>" % (index_inspect.url,host))
except requests.exceptions.Timeout as e:
logger.warn('Network timeout trying to reach host %s, this may be due to firewall restrictions or an issue with the service we are trying to contact.', host)
body.append("WARNING: Host %s is unreachable! <br>" % (host))
return
# Correctly splits and formats the text we get from the index_language HTTP response
langreader = csv.DictReader(index_language.text.split("\n"),
delimiter=",",
fieldnames=['key', 'description', 'format', 'level', 'valueMax'])
# For loop over our new list of meta and update metakey and metamax values from it into an array.
for meta in langreader:
metakey = meta['key']
metamax = meta['valueMax']
# Iterate over JSON we get from index_inspect page
inspectreader = json.loads(index_inspect.text)
for line in inspectreader['params']:
metavalues = line.get('pages')
if str(line.get('key')) == str(metakey):
if metavalues > "0" and metamax > "0":
if metamax.isdigit():
percent = int(metavalues) * 100 / int(metamax)
#DEBUG purposes print str(metakey) + " : " + str(metamax) + " : " + str(metavalues) + " : " + str(percent) + "% Used"
###### Call warning function ######
warning(metakey,percent,host,metamax,metavalues)
##############################################
## Function to check values and build array ##
##############################################
def warning(metakey,percent,host,metamax,metavalues):
metakey = str(metakey)
metamax = str(metamax)
metavalues = str(metavalues)
if percent >= 100:
percent = str(percent)
body.append("%s, Max Value: %s, Actual Value: %s, %s percent over <br>" % (metakey, metamax, metavalues, percent))
logger.info('%s : %s : %s : %s % Used', metakey, metamax, metavalues, percent)
#################################################
## Function to build email message and send it ##
#################################################
def email(body):
msg = MIMEMultipart('alternative')
msg['Subject'] = 'Max index report - ' + d1 + ' ' + t1
msg['From'] = sender
msg['To'] = receiver
msg.preamble = 'Max index report'
html = """\
<html>
<head><b> Max Index Report </b></head>
<body>
<p>
<b> Location: /root/scripts/check-indexes/index_profile.py </b>
<br>
<b> Report Generated on %s at %s </b>
%s
<br>
</p>
</body>
</html>
""" %(d1, t1, ''.join(body))
message = MIMEText(html, 'html')
msg.attach(message)
try:
logger.info('Creating SMTP object...')
smtpObj = smtplib.SMTP('internal.smtp.server.domain.com')
logger.info('Send the mail object over to default SMTP server...')
smtpObj.sendmail(sender, receiver, msg.as_string())
logger.info('Successfully sent email')
except SMTPException:
logger.warn('Error: unable to send email')
## Run main ##
if __name__=='__main__':
main()
4 REPLIES 4
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2016-12-30 05:46 AM
Do lines 176 -186 need to be further indented as otherwise it only interates over the last metakey?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2017-01-03 04:01 PM
No actually. The first call to the warning function on line 177/178 is inside the for loop for each retrieved meta value, and thus it'll keep calling it for each instance identified. As the function just takes a few parameters (single metakey) and does the check, it then appends that body object and moves on.
If you run into an issue with it, please let me know...
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2017-01-26 04:00 PM
David, I ran into some problems with my script, I was concerned I wasn't getting what I expected to see.
I did some more tuning/tweaking of the code, and now I think I'm getting what I want, artificially lowered all the index valueMax settings on several keys on my dev environment and then ran it to check.
I'll post my update to my code shortly when I am more confident.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2017-01-26 06:06 PM
Here is the most up to date version of the script.
#!/usr/bin/env python
# Author: kddiens
# Changelog:
# 03/29/2016 - kdd - created off sample code from RSA, added email function and formatting
# 03/30/2016 - kdd - restructing into a loop for main so we can go across multiple hosts
# 03/31/2016 - kdd - fixes and cleanup, adding script location to html output
# 12/21/2016 - kdd - Adding internal script logging to ease updating, writing, etc
###########################################################################################
import requests, csv, sys, argparse, json, smtplib, time, datetime, os, logging
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
########################
### Global Variables ###
########################
sender = 'sample@domain.com'
receiver = 'sample@domain.com'
today = datetime.datetime.today()
now = datetime.datetime.now()
t1 = now.strftime("%H:%M:%S")
t2 = now.strftime("%H")
d1 = today.strftime("%Y-%m-%d")
body = []
hosts = []
filename = os.path.join("/root/scripts/applianceList/concentratorlist")
log_file = '/root/scripts/check-indexes/index_profile_concentrators' + str(d1) + '_' + str(t2) + '.log'
user = 'serviceaccountname'
password = 'SECRET PASSWORD'
port = "50105"
language_payload = { 'msg' : 'language', 'force-content-type' : 'text/plain', 'expiry' : '600' }
inspect_payload = { 'msg' : 'inspect', 'force-content-type' : 'application/json', 'expiry' : '600' }
###############
### Logging ###
###############
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
#set debug logging
#logging.basicConfig(level=logging.DEBUG)
#logger.setLevel(logging.DEBUG)
# create file handler for logger
handler = logging.FileHandler(log_file)
handler.setLevel(logging.INFO)
# create logging format
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
# add the handles to the logger
logger.addHandler(handler)
###############
# MAIN #
###############
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--host", help="Host IP or Name", dest="host", action="store")
parser.add_argument("--file", help="Input File", dest="fflag", action="store_true")
args = parser.parse_args()
if not args.host and not args.fflag:
logger.info('Hostname not specified and file flag not set, unable to continue...')
sys.exit("Hostname not specified with --host and input file flag not defined via --file")
if args.fflag and args.host:
logger.info('Hostname and file flag not null, these are exclusive arguments, unable to continue..')
sys.exit("--file and --host parameters are exclusive, they cannot be combined, try again")
if args.host:
logger.info('Hostname exists, passing param to function for individual host %s...' % (args.host))
body.append("<br>The following metakeys on <b>%s</b> are over used and require index config custom updates: <br>" % (args.host))
#collect_meta(args.host,args.ssl) potential to add SSL flag during runtime to toggle if HTTP or HTTPS is used
collect_meta(args.host)
elif args.fflag:
logger.info('File flag set, using file declared in global variable %s...' % (filename))
logger.info('Changing state to build_hosts() function.')
build_hosts()
filtered_hosts = filter(None, hosts)
for host in filtered_hosts:
body.append("<br>The following metakeys on <b>%s</b> are over used and require index config custom updates: <br>" % (host))
#collect_meta(host,args.ssl)
collect_meta(host)
email(body)
############################################################
## Read in concentrator list line by line and create list ##
############################################################
def build_hosts():
logger.info('Attempting to open file %s to build hosts', filename)
f = open(filename, 'r')
for line in f.read().split('\n'):
logger.info('Adding host %s from file into hosts array', line)
hosts.append(line)
f.close()
###########################################
## Function to collect the meta per host ##
###########################################
def collect_meta(host):
logger.info('Host passed to function is %s', host)
# Construct the URI, will add path later
host_uri = "http://" + host + ":" + port
logger.info('Host URI constructed is %s', host_uri)
# Do a try catch and get a HTTP response from the system
try:
index_language = requests.get(host_uri + '/index',
verify=False,
auth=(user, password),
params=language_payload,
timeout=5)
logger.info('Full URL to locate index langauge is %s', index_language.url)
if index_language.status_code == 200:
logger.info('w00t! HTTP 200 response!')
elif index_language.status_code != 200:
logger.warn('Program received status code %s for host %s using URL %s', index_language.status_code, host, index_language.url)
body.append("WARNING: Cannot reach URL: %s on host %s <br>" % (index_language.url,host))
except requests.exceptions.Timeout as e:
logger.warn('Network timeout trying to reach host %s, this may be due to firewall restrictions or an issue with the service we are trying to contact.', host)
body.append("WARNING: Host %s is unreachable! <br>" % (host))
return
try:
index_inspect = requests.get(host_uri + '/index',
verify=False,
auth=(user, password),
params=inspect_payload,
timeout=5)
logger.info('Full URL to locate index inspect page is %s', index_inspect.url)
if index_inspect.status_code == 200:
logger.info('w00t! HTTP 200 response!')
elif index_inspect.status_code != 200:
logger.warn('Program received status code %s for host %s using URL %s', index_inspect.status_code, host, index_inspect.url)
body.append("WARNING: Cannot reach URL: %s on host %s <br>" % (index_inspect.url,host))
except requests.exceptions.Timeout as e:
logger.warn('Network timeout trying to reach host %s, this may be due to firewall restrictions or an issue with the service we are trying to contact.', host)
body.append("WARNING: Host %s is unreachable! <br>" % (host))
return
# Correctly splits and formats the text we get from the index_language HTTP response
langreader = csv.DictReader(index_language.text.split("\n"),
delimiter=",",
fieldnames=['key', 'description', 'format', 'level', 'valueMax'])
# For loop over our new list of meta and update metakey and metamax values from it into an array.
inspectreader = json.loads(index_inspect.text)
# Loop over the sequence of dictionaries (the langreader page) and then loop again
for row in langreader:
# loop over each dictionary in the inspect page comparing it with the key in the langreader sequences
for line in inspectreader['params']:
# set the metakey as the key in the dictionary in inspectreader sequence
# also sets the metavalues from the inspectreader sequence, note we use pages instead of values, since values will max out and always show 100 percent
metakey = line.get('key')
metavalues = line.get('pages')
# if the metakey in the langreader sequence matches with the inspectreader sequence do some stuff
if metakey == row['key']:
# we assign the metamax var here because otherwise we'll grab the wrong max value from langreader sequence instead
metamax = row['valueMax']
# verify we don't have some extra sequence with None as the value
if metavalues is not None and metamax is not None:
#convert the str and unicde types to integers to do basic arithemtic on it
metavalues = int(metavalues)
metamax = int(metamax)
if metavalues > 0:
if metamax > 0:
percent = metavalues * 100 / metamax
warning(metakey,percent,host,metamax,metavalues)
##############################################
## Function to check values and build array ##
##############################################
def warning(metakey,percent,host,metamax,metavalues):
metakey = str(metakey)
metamax = str(metamax)
metavalues = str(metavalues)
if percent >= 100:
percent = str(percent)
body.append("%s, Max Value: %s, Actual Value: %s, %s percent over <br>" % (metakey, metamax, metavalues, percent))
logger.info('%s : %s : %s : %s % Used', metakey, metamax, metavalues, percent)
#################################################
## Function to build email message and send it ##
#################################################
def email(body):
msg = MIMEMultipart('alternative')
msg['Subject'] = 'Max index report - ' + d1 + ' ' + t1
msg['From'] = sender
msg['To'] = receiver
msg.preamble = 'Max index report'
html = """\
<html>
<head><b> Max Index Report </b></head>
<body>
<p>
<b> Location: /root/scripts/check-indexes/index_profile.py </b>
<br>
<b> Report Generated on %s at %s </b>
%s
<br>
</p>
</body>
</html>
""" %(d1, t1, ''.join(body))
message = MIMEText(html, 'html')
msg.attach(message)
try:
logger.info('Creating SMTP object...')
smtpObj = smtplib.SMTP('internal.smtp.server.domain.com')
logger.info('Send the mail object over to default SMTP server...')
smtpObj.sendmail(sender, receiver, msg.as_string())
logger.info('Successfully sent email')
except SMTPException:
logger.warn('Error: unable to send email')
## Run main ##
if __name__=='__main__':
main()
