HTTP Lua Parser Options File

Caution: RSA strongly suggests that you do not subscribe to the options file. Subsequent downloads of this file will overwrite all changes that you have made to the file.

Note the following:

  • If you deploy the options file, it can be found in the same directory as parsers: /etc/netwitness/ng/parsers/.
  • The parser is not dependent upon the options file. The parser will load and run even in the absence of the options file. The options file is only required if you need to change the default settings.
  • If you do not have an options file (or if your options file is invalid), the parser uses the default settings.

Note: The parser will never use both the defaults and customized options. If the options file exists and its contents can be loaded, then the defaults will not be used at all.

The HTTP_lua_options file contains the following options for controlling the parser:

  • registerURL
  • splitQuery
  • useOrigIP
  • refererPath
  • userAgent
  • respReason
  • decompress
  • advanced
  • customHeaders

To change an option from false to true, edit the line inside the corresponding function, from

return false

to

return true

And similarly to go from true to false.

Note: Modifying any of these options requires a service restart to take effect; a simple parser reload is insufficient.

registerURL

Default value: false

Default behavior is for the host, directory, filename, extension, and query to be registered as separate meta values with the appropriate keys:

  • alias.host: www.example.com
  • directory: /someDir/
  • filename: somefile.html
  • extension: html
  • query: ?foo=bar

This option instead registers them as a single meta value (individual keys will not be registered due to redundancy): url: www.example.com/someDir/someFile.html?foo=bar

Note: The registered URL is a reconstructed approximation: it may not be the exact URL that was clicked on.

splitQuery

Default value: false

Default behavior is for the entire query string from a request to be registered as a single meta value:

query: alpha=one&beta=two&gamma=three

If this option is enabled, then each element of a query string will be registered as individual meta values:

query: alpha=one
query: beta=two
query: gamma=three

useOrigIP

Default value: true

Default behavior is to register values from x-forwarded-for headers and the like with index key orig_ip. If this option is disabled, then values are registered as following:

  • hostnames: "alias.host"
  • IPv4: "alias.ip"
  • IPv6: "alias.ipv6"
  • email address: "email"
  • other: "alias.host"

refererPath

Default value: false

Default behavior is to register the value of a "Referer:" header as "referer" meta. If this option is enabled, then the host, directory, filename, extension, and querystring values are broken out from Referer and registered individually. In order to avoid duplication, the entire Referer value will not be registered.

For example, assume the following header:

Referer: http://www.example.com/hello/world.html?foo=bar&one=two

If this option is disabled (default), then the following meta is registered:

Referer: http://www.example.com/hello/world.html?foo=bar&one=two

If enabled, then the following meta is registered:

alias.host: www.example.com
directory: /hello/
filename: world.html
extension: html
query: foo=bar&one=two

Note that if the Split Query String option is also enabled, then the query string is registered individually (see splitQuery above).

userAgent

Default value: client

Default behavior is to register the value of User-Agent headers with the client index key.

Modifying this value causes User-Agent values to additionally be registered with the specified key. If the key does not already exist it will be created (normal key name restrictions apply).

Note that this results in duplication of meta. User-agent is registered to both client and the specified key.

respReason

Default value: true

For response codes other than 2xx, default behavior is to register both the status code and reason phrase together as error meta. For example: error: 404 Not Found.

Disabling this option (setting to false) causes only the response code to be registered. For example: error: 404

decompress

Default value: 0 (means that decompression will not be performed for any content type, which maximizes performance)

Decompress content-encoded HTTP responses. Encodings gzip, deflate, and chunked are supported. Enabling this option provides visibility into such responses to other parsers. Decompression incurs a performance penalty which varies depending upon the prevalence of compressed or encoded HTTP responses seen in the environment. This can be ameliorated to some extent by choosing to only decompress specific content types.

This is a bit-packed value representing the content types to decompress, where:

  • 0 Decompress OFF
  • 1 application/*
  • 2 audio/*
  • 4 font/*
  • 8 image/*
  • 16 message/*
  • 32 model/*
  • 64 text/*
  • 128 video/*
  • 255 Decompress ALL content types

Just add desired values together. For example:

  • To decompress application (1) and text (64), use 65 (1+64)
  • To decompress application (1), audio (2), and text (64), use 67 (1+2+64)
  • To decompress font (4) and image (8), use 12 (4+8)

The default value of 0 means that decompression will not be performed for any content type, which maximizes performance. A value of 65 specifies that content-types "application" and "text" will be decompressed. This should provide a good balance of visibility and performance. To maximize visibility, a value of 255 will enable decompression of all content types.

Enabling a content-type enables all constituent sub-types. For example, "application" includes "application/octet-stream", "application/javascript", and so on.

NOTES:

  • Only valid for versions 11.0 and newer. This option has no effect on versions 10.x or older as they do not have the capability to decompress encoded HTTP responses.
  • Has no effect on instances of compression which are not HTTP responses, such as compressed archive files (such as ZIP and RAR), LZMA streams, and so on.
  • If enabled, then 65 is the suggested value (not 255). That will decompress data that is typically most interesting to analysts (web pages, executables, javascript), without wasting resources decompressing larger files that are typically less interesting (images, audio, video, and so forth).

advanced

Default value: false

If this parameter is enabled, the system performs advanced analysis of HTTP characteristics. Analysis includes only the first request and first response. Meta is registered to the key analysis.service.

customHeaders

Provides a mechanism for customers to register HTTP header values to specified keys.

Meta registered will be in addition to—not replacement of—standard meta registration. For example, if you specify user-agent headers be registered to key foo, user-agent will still also be registered to alias.host (or alias.ip/alias.ipv6 if appropriate).

Therefore, beware of registering too much data, which could lead to excessive duplication and thus impact performance and data retention.

Syntax:

["header"] = "key",

Where:

  • header is the desired HTTP header in lowercase. Do not included spaces, colons, and so forth.
  • key is the desired meta key with which to register the value of header

Notes:

  • Meta key names can be up to 16 characters long, and can contain only letters or the '.' (period) character. Keys specified that do not meet these requirements may be modified in order to conform.
  • Keys are registered as format="Text". Do not use keys indexed in other formats.