(Optional) Configure Hash Filter

This topic introduces hash filters as a method of marking files in Malware Analysis that are known to be good or known to be bad. Hash filtering allows you to maintain a list of known good or known bad file hashes. In the Hash tab, you can fine tune Malware Analysis event analysis based on file hashes. When a file hash is marked as Good, Malware Analysis does not analyze the file the next time it is seen. When a file hash is marked as Bad, Malware Analysis automatically raises the file’s community score by a large number of points. Malware Analysis still analyzes the file, just in case new information becomes available.

Note: If an event contains a single file and that file’s hash is marked as Good, Malware Analysis filters the entire event and you do not see it in Malware Analysis results.

To add hash filters to the hash list, you can use either of these manual methods:

  1. Context menu add in the Event Detail view: Right-click on a file, and a context menu allows marking of the hash for the selected file as Good (Normal) or Bad (Malicious).
  2. Hash tab toolbar: Click Add in the Hash tab to add a file hash, file size, and optionally, mark the hash as trusted.

There is also an automated method to add hash filters to Malware Analysis by importing a hash list in bulk from the watched folder. Hashes imported through the watched folder do not appear in the hash list. With bulk importing and the watched directory (/var/netwitness/malware-analytics-server/spectrum/hashWatch) on the Malware Analysis server set up, copy a hash list into the watched folder to be automatically imported into the system. Hashes imported using the bulk import method overwrite hashes that were previously imported through the watched folder.

View the Hash List

To view the Hash List:

  1. Go to netwitness_adminicon_25x22.png (Admin) > Services.
  2. In the Services view, select a Malware Analysis service, and select netwitness_ic-actns.png > View > Config.

  3. Select the Hash tab.

    The hash list is displayed in the Hash tab. Only file hashes that have been added using one of the methods are displayed.

Add a File Hash to the Hash Filter

To add a file hash to the hash filter:

  1. In the Hash tab, in the toolbar, click Add.

    The Add Hash dialog is displayed.

  2. If the hash is trusted, select Trusted.
  3. Enter the MD5 hash and the file size in bytes.

  4. Click Save.

    The file hash is added to the hashes and used to perform hash filtering in Malware Analysis.

Mark a Hash as Trusted or Untrusted

To mark a file hash as trusted or untrusted:

  1. In the Hash tab, to toggle between trusted and untrusted, click in the Trusted column for the hash.
  2. In the toolbar, click Save Edit.

Delete a Hash from the Hash Filter

To delete a hash from the hash filter:

  1. In the Hash tab, select one or more hashes that you want to remove from the hash filter.

  2. In the toolbar, click Delete.

    A dialog requests confirmation and offers an opportunity to cancel.

  3. To confirm the deletion, click Yes.

    The file hash is deleted from the grid and no longer used to perform hash filtering in Malware Analysis.

Search for a File Hash

In the Hash tab, you can search for a file hash that is displayed in the grid. In the MD5 field, type the file hash for which you are searching, and click Search. The list of files that contain the hash is displayed in the grid.

Import a Hash List Using the Watched Folder

To import a hash list from the watched directory, the hash list must be in the specified format and must be sorted on md5. You can drop a file formatted as described below into a folder (/var/netwitness/malware-analytics-server/spectrum/hashWatch) on the Malware Analysis appliance, and it is automatically imported into the local hash database. This is the only way to import file hashes into. An additional use case is to allow a system administrator to expose the watched directory to some process that would push a file to this directory. This is a bulk import method designed to handle a high volume of hash imports.

This is a csv-formatted file with no spaces between the data in each row. The assumption with the data in the hash list is that there are no duplicates. Duplicates are ignored during processing. If duplicate hashes are encountered, the log file will display the following message to indicate the number of duplicate hashes contained in the file:

2013-08-09 09:46:00,674 [jobExecutor-2(HashFileWatch)] INFO com.netwitness.malware.core.scheduler.jobs.HashFileWatch - Processing - /var/lib/rsa>malware/hashWatch/test.csv
2013-08-09 09:47:56,619 [jobExecutor-2(HashFileWatch)] INFO com.netwitness.malware.core.services.file.hash.HashServiceImpl - Skipped 21 Duplicate Hashes Already on File
2013-08-09 09:48:06,638 [jobExecutor-2(HashFileWatch)] INFO com.netwitness.malware.core.scheduler.jobs.HashFileWatch - Processed - / var/lib/rsa>malware/hashWatch/test.csv

Below is an example of a hash list in the default file format.


[BeginFileExample]
392126E756571EBF112CB1C1cdEDF926,98865,True
0E53C14A3E48D94FF596A2824307B492,2226,True
176308F27DD52890F013A3FD80F92E51,42748,False
9B3702B0E788C6D62996392FE3C9786A,32768,False
937ADE76A75712B7FF339403B4FCB5A6,4821,False
B47139415F735A98069ACE824A114399,1723,False
E6CAF205E602CFA9A65663DB1A087874,704,False
680CA0BCE1FC7BC4136ADF4E210869C5,2075,False
[EndFileExample]


A NetWitness configuration file (/var/netwitness/malware-analytics-server/spectrum/conf/hashFileWatchConfig.xml) specifies the format and options in the hash list import process. Below is a listing of the configuration file.

<config>
<enabled>true</enabled>
<distributedCacheEnabled>true</distributedCacheEnabled>
<watchDirectory>//var/lib/rsamalware/hashWatch</watchDirectory>
<processedDirectory>/var/lib/rsamalware/hashWatch/processed</processedDirectory>
<erroredDirectory>/var/lib/rsamalware/hashWatch/error</erroredDirectory>
<md5Col>0</md5Col>
<fileSizeCol>-1</fileSizeCol>
<isTrustedCol>1</isTrustedCol>
<isTrust>false</isTrust>
<ignoreFirstLine>false</ignoreFirstLine>
<frequencyInMinutes>1</frequencyInMinutes>
<isGzipCompressed>false</isGzipCompressed>
</config>
LineDescription

<md5Col>0</md5Col>

The location of the md5 hash in each entry. The default value is position 0, or the first position.

<fileSizeCol>1</fileSizeCol>

The location of the hash size in each entry. The default value is position 1, or the second position. If the hash size is not included in the csv file, the value must be -1.

<isTrustedCol>2</isTrustedCol>

The location of the Trusted Column in each entry. The default value is position 2. If the Trusted parameter is not included in the csv file, the value must be -1.

<isTrust>false</isTrust>

The default assumption for Trusted in each entry is false.

<ignoreFirstLine>false</ignoreFirstLine>

The presence or absence of a header in the hash. The default value is false. If the hash has a header, the value must be set to true.

<frequencyInMinutes>1</frequencyInMinutes>

The interval between checks by NetWitness in the watched directory. The default value is 1 minute.

<isGzipCompressed>false</isGzipCompressed>

The hash is compressed using Gzip. The default value is false. If the hash is Gzip compressed, the value must be set to true here.

When the hash list has been imported, the system log has entries similar to this:

2013-04-11 03:22:00,597 [jobExecutor-9(HashFileWatch)] INFO com.netwitness.malware.core.scheduler.jobs.HashFileWatch - Processing - /var/lib/rsamalware/spectrum/hashWatch/simpleHash.csv
2013-04-11 03:22:00,600 [jobExecutor-9(HashFileWatch)] INFO com.netwitness.malware.core.scheduler.jobs.HashFileWatch - Processed - /var/lib/rsamalware/spectrum/hashWatch/simpleHash.csv

If there is a problem loading the file, the system log has entries similar to this:

2013-04-11 03:17:00,597 [jobExecutor-4(HashFileWatch)] INFO com.netwitness.malware.core.scheduler.jobs.HashFileWatch - Processing - /var/lib/rsamalware/spectrum/hashWatch/simpleHash.csv
... Verbose log
2013-04-11 03:17:00,632 [jobExecutor-4(HashFileWatch)] INFO com.netwitness.malware.core.scheduler.jobs.HashFileWatch - Error Processing - /var/lib/rsamalware/spectrum/hashWatch/simpleHash.csv

To import a hash list using the watched folder method:

  1. Copy the hash lists that you want to import into the /var/netwitness/malware-analytics-sever/spectrum/hashWatch directory.
    Malware Analysis automatically watches this folder and processes files placed there.
    Malware Analysis adds every hash found in the hash lists to the hash filter.
    If there are processing errors, they are logged in /var/netwitness/malware-analytics-sever/spectrum/hashWatch/error
    Processed files are cataloged in /var/netwitness/malware-analytics-sever/spectrum/hashWatch/processed
    Processed files are not removed from the hashWatch directory.
  2. After importing hashes in bulk, the System Administrator can use a cronjob to clean up old processed files.