Decoder Commands for Managing Rules

In the NetWitness Core database, the Rules tree holds the main functionality related to managing rules for all Core services that have rules: Concentrators, Decoders, Log Decoders, and Archivers. Although you can manage rules in the NetWitness user interface, advanced users may prefer to manage rules using a command line to add, merge, replace, delete, and validate rules on a service. This section provides a brief overview of the commands and their usage. These are the available commands:

  • add - Adds a single rule at the specified position.
  • clear - Deletes all existing rules in the current node on the service. For example, using the command in /decoder/config/rules/application node deletes all existing application rules on the Decoder.
  • delete - Deletes one or more rules at a specified position and count.
  • merge - Merges a pushed rule set with an existing rule set. Existing rules that match the incoming rules (by name or rule) are replaced; otherwise, rules are inserted by the position indicated as described in merge Command.
  • replace - Deletes all existing rules and replaces them with the incoming rule set.
  • validate - Validates the syntax of a rule, but does not validate the meta keys.

add Command

The add command adds the rule to the existing rule set. Formatting is important because the API uses double quotes in the rule language and also uses double quotes as parameters to all NetWitness APIs. Therefore, you must escape any double quotes in the rule itself by preceding it with a backslash (\) character. This is the syntax of the command:

add rule=<string> name=<string> alert=<string, optional> atPos=<uint32, optional>

  • rule is the rule to add. Be sure to place double quotes around any rules with white space and to escape any double quotes that are part of the rule with a backslash.
  • name is the name of the rule.
  • alert is the alert for the rule (if any).
  • atPos is the position at which the rule should be added (1 based). Zero is the top of the list and any number larger than the current size of the list is appended to the list.

This is an example of command to add a rule using NwConsole
send /decoder/config/rules/application add rule="ip.src exists" order=1 alert=ioc name=testrule

For example, take the following rule:

alias.host = "myPC" && country.src="china","russian federation"

To add this as a rule, you would need to send the parameters as follows:

rule="alias.host = \"myPC\" && country.src=\"china\",\"russian federation\"" name=myRule filter

Notice how all the double quotes had to be escaped inside the rule parameter. A simple trick to make this more readable is to use single quotes inside the rule. Single and double quotes are interchangeable in the rule and query language, but not in parameters for the API (only double quotes are supported there). Therefore, this is more readable:

rule="alias.host = 'myPC' && country.src='china','russian federation'" name=myRule filter

merge Command

The merge command is used to merge an incoming list of rules with the existing rules on the service. This is how it works:

  • It finds existing rules that match via the name OR via a matching rule, updates the existing rule name, and keeps the same position.
  • It inserts new rules into the rule list based on the NUMBER position. If the number is zero, it goes to the top of the list.
  • It processes the rules in the order received so if you have two rules numbered zero, the second rule is processed after the first and claims the top spot. All existing rules are pushed down two places. Any numbers higher than existing rule positions are appended after the last existing rule and numbered in sequence.
  • Any non-numbered rule is appended after the last existing rule and numbered in sequence.

This is the syntax of the merge command:

merge --file-data=<string> --file-format<string>

  • file-data is the full path and name of the rules file to merge.
  • file-format is the format of the rules file. Valid values are params-list, string, params, binary, and params-binary.

Methods of Sending a List of Rules to a Service

There are two ways to send a list of rules. You can send them as a .nwr (NetWitness Rule) file or as a numbered set of parameters, each number indicates the position to insert the rule at as well as the encoded rule. If you want to see the current list of rules on a service, you need to run the ls command on the rule category (for instance, application rules on a Decoder are found in /decoder/config/rules/application).

This is an example of commands to list the existing rules using NwConsole:

login <hostname>:50004 <username> <password>

cd /decoder/config/rules/application

ls

This is another example to list existing rules in NwConsole:

send /decoder/config/rules/application ls

This is an example of the command to point to network rules in the RESTful port, which supports a basic admin HTML app.

http[s]://<decoder>:50104/decoder/config/rules/network

Send a NetWitness Rule File

Let's start with an example nwr file, each rule must be on a separate line:

rule="ip.src=192.168.0.1" name=first keep

rule="ip.src=192.168.1.1" name=second alert=ioc

rule="ip.src=192.168.2.1" name=third filter

To push and merge rules using NwConsole, use the following commands:

login <hostname>:50004 <username> <password>

send /decoder/config/rules/application merge --file-data=/root/App_Rules.nwr --file-format=params-list

To replace the existing rules with the rules in the file, instead of using the merge command, use the replace command.

send /decoder/config/rules/application replace --file-data=<pathname> --file-format=params-list

To merge the rules in an nwr file using the RESTful port, you can use a curl command that pushes the rules:

curl -u "<username>:<password>" -H "Content-Type: application/octet-stream" --data-binary @<pathname> -X POST "http://<hostname>:50104/decoder/config/rules/application?msg=merge"

The examples are pushing application rules. To push network rules, send the rules to /decoder/config/rules/network. For correlation rules, send the rules to /decoder/config/rules/correlation.

Send Numbered Parameters

The other way to send a list of rules is to send them as numbered parameters. The difficulty with this method is remembering to escape the quotes within each numbered rule. Though it is only a problem if you are trying to do it by hand. For instance, to send the same rules above as parameters via NwConsole, use the following command:

send /decoder/config/rules/application merge 1="rule=\"ip.src=192.168.0.1\" name=first keep" 2="rule=\"ip.src=192.168.1.1\" name=second alert=ioc" 3="rule=\"ip.src=192.168.2.1\" name=third filter"

This command is hard to read because you have to escape the inner quotes with a backslash (\). Otherwise, these two commands accomplish the same thing. Merging or adding three rules in positions 1, 2 and 3. If you think the above was hard to read, this is what the equivalent curl command looks like:

curl -u "<username>:<password>" "http://<hostname>:50104/decoder/config/rules/application?msg=merge&1=rule%3D%22ip.src%3D192.168.0.1%22%20name%3Dfirst%20keep&2=rule%3D%22ip.src%3D192.168.1.1%22%20name%3Dsecond%20alert%3Dioc&3=rule%3D%22ip.src%3D192.168.2.1%22%20name%3Dthird%20filter"

For more details on how to escape double quotes inside parameters, see add Command.

Ordering Rules When Pushing

Pushed rules are ordered in one of two ways. When passing as parameters, the number of each parameter determines the insertion order. If it is not actually a number, merge checks for an order parameter within the rule itself and uses that value if found.

Note: Using order is the only way to set the order with a .nwr file. If neither a number nor an order parameter is found, there are no guarantees of the insertion order.

Example

A Decoder has the following application rules installed; notice the numbering is ALWAYS consecutive and starts at 1:

0001 : rule="ip.src = 192.168.0.1 || ip.dst = 192.168.0.1 || alias.host = 'My-PC'" name=first keep

0002 : rule="ip.src=192.168.1.1" name=second alert=ioc

0003 : rule="ip.src=192.168.2.1" name=third filter

And you want to merge the following four rules:

rule="ip.src=192.168.3.1" name=third keep

rule="ip.dst=192.168.4.1" name=NewRule filter order=0

rule="alias.host = 'pc1','pc2'" name=filterTheseNames filter order=append

rule="service=80,443" name=web filter order=3

Use any method to push your rules and this is what you end up with:

0001 : rule="ip.dst=192.168.4.1" name=NewRule filter order=1

0002 : rule="ip.src = 192.168.0.1 || ip.dst = 192.168.0.1 || alias.host = 'My-PC'" name=first keep order=2

0003 : rule="service=80,443" name=web filter order=3

0004 : rule="ip.src=192.168.1.1" name=second alert=ioc order=4

0005 : rule="ip.src=192.168.3.1" name=third keep order=5

0006 : rule="alias.host = 'pc1','pc2'" name=filterTheseNames filter order=6

Are there any surprises here? This is how each rule was processed.

1. rule="ip.src=192.168.3.1" name=third keep

This rule had the same name as an existing rule on the Decoder (third). So the rule updated the existing rule, changing _filter_ to _keep_.

2. rule="ip.dst=192.168.4.1" name=NewRule filter order=0

This rule is new and had order=0 in it, which means insert at the very top.

3. rule="alias.host = 'pc1','pc2'" name=filterTheseNames filter order=append

This rule had a non-number append for order, therefore, it went to the end of the list. You can accomplish the same thing by giving a very large number, like 999999.

4. rule="service=80,443" name=web filter order=3

This rule is last but has order=3, therefore, if it does not match an existing rule by name or the text of the rule itself, it should be placed in position 3. And there it is, the third rule in the list. Any rules that follow were pushed further down.

replace Command

The replace command removes all existing rules and replaces them with the incoming rule list. Refer to merge Command for details on how to format the incoming rule list and how ordering works.

This is an example of the replace command using a NetWitness Rule File :

send /decoder/config/rules/application replace --file-data=/root/Decoder-AppRules.nwr --file-format=string

This is an example of the replace command using Numbered Parameters :

send /decoder/config/rules/application replace 1="rule=\"ip.src exists\" name=\"test rule\" order=1 alert=ioc"

clear Command

The clear command removes all existing rules on the service. This is an example of the command:

send /decoder/config/rules/application clear

delete Command

The delete command deletes one or more rules on the service.

delete atPos <uint32> count <uint32, optional>

  • atPos deletes the rule at the given position. Rules are numbered starting with 1 and go in sequential order.
  • count deletes one or more rules starting atPos. This is an optional parameter defining the number of rules to delete starting atPos. The default value is 1.

This example of the command deletes four rules beginning at position 0003:

send /decoder/config/rules/application delete atPos=0003 count=4

validate Command

The validate command takes the provided rule and verifies that it parses correctly. Keep in mind that this command cannot verify whether language keys and entities are valid.

validate rule <string>

rule - is the name of the rule to validate. Make sure to place double quotes around any rules with white space.