Transitioning to Lua Parsers

[Anonymous]

Is there any matrix of the new lua parsers and the old parser(s) they replace?  Some of them are obvious, others less so.  Anybody have experience switching over?

[Anonymous]

Our product engineer and Lua expert said he'd get you some more substantive detail later in the day but for now he had some quick notes:

* most parsers have the same name, just with "_lua" appended

* a few were renamed for clarity, e.g. "pkware" -> "fingerprint_zip"

* a few were combined into a single lua parser, e.g. "fingerprint_jar" and "fingerprint_java_class" were combined into "fingerprint_java"

[RSAAdmin]

Not all of the listed lua parsers are in Live yet, but will be.

LUA PARSER	REPLACES FLEX OR NATIVE PARSER(S)
AIM_lua	aim_oscar
dr_watson_lua	dr_watson
BGP_lua	bgp
bittorrent_lua	bittorrent, bittorrent-id, fingerprint_bittorrent, BITTORRENT
botnet_lua	botnet
creditcard_detection_lua	creditcard_detection
db2_lua	db2
DNP3_lua	dnp3
DNS_verbose_lua	DNS, dns_verbose
duqu_lua	duqu
ein_detection_lua	ein_detection
ethernet_oui	MAC_Vendor
fingerprint_access_db_lua	fingerprint_access_db
fingerprint_apple_dmg_lua	fingerprint_apple_dmg
fingerprint_appleExec_lua	fingerprint_apple_exec
fingerprint_apple_ios_lua	fingerprint_apple_ios
fingerprint_apple_iwork_lua	fingerprint_apple_iwork
fingerprint_cab	fingerprint_cab_files
fingerprint_cad_lua	fingerprint_cad
fingerprint_chm_lua	fingerprint_chm, malware_chm
fingerprint_flash	fingerprint_swf, base64_swf
fingerprint_gif_lua	fingerprint_gif
fingerprint_java	fingerprint_jar, fingerprint_java_class
fingerprint_javascript_lua	fingerprint_javascript, javascript, javascript_suspicious, javascript_packers, javascript_shellcode
fingerprint_jpg_lua	fingerprint_jpg
Fingerprint_Private_Key	fingerprint_private_encryption_keys
fingerprint_lnk_lua	fingerprint_lnk, exploit_lnk_file
fingerprint_msi_lua	fingerprint_msi
fingerprint_mssql_lua	fingerprint_mssql
fingerprint_office_lua	fingerprint_office95-2003, fingerprint_office_2007, encoded_file_fingerprinting
fingerprint_pdf_lua	fingerprint_pdf, malware_pdf, malware_pdf_v201
fingerprint_php_lua	fingerprint_php
fingerprint_key	fingerprint_pkcs12
fingerprint_png_lua	fingerprint_png
fingerprint_rar_lua	fingerprint_rar
fingerprint_rtf_lua	fingerprint_rtf, encoded_file_fingerprinting
fingerprint_unix_script_lua	fingerprint_unix_script
fingerprint_zip	pkware
FIX_lua	FIX
Form_Data_lua	Form_Data_Elements
ghost	ghost_protocol
gnutella_lua	GNUTELLA
htran_lua	htran
HTTP_lua	HTTP, HTTP-flex, http_connect, http_error_codes, NTLMSSP, crafted_http_header, http_header, xfwdfor, ICAP_HTTP
HTTP_SQL_Injection	http_sql_injection
IMAP_lua	IMAP, IMAP-flex
IRC_verbose_lua	irc, irc-expanded
iSCSI	iscsi
MAIL_lua	MAIL, MAIL-flex, email-ip
modbus	modbus-w_port
NFS_lua	NFS, nfs-flex, sunrpc
NTLMSSP_lua	NTLMSSP
ntp_lua	NTP
OCSP_lua	OCSP
Packers	packers
phishing_lua	phishing, email_url_host
pwdump	encoded_hashes
QQ_lua	qq
RDP_lua	RDP
ripng_lua	ripng
rtmp_lua	RTMP
shadyrat_lua	shadyrat
SMB_lua	SMB, SMB-flex, SMB-ID
socks_lua	socks
SoulSeek_lua	SoulSeek
spectrum_lua	spectrum, spectrum11
SSH_lua	SSH
TDS_lua	TDS
TLD_lua	TLD
TLS_lua	TLSv1, TLS-flex, TLS_id
TN3270E_lua	tn3270e
VNC	vnc-rfb
windows_command_shell_lua	SHELL, windows_command_shells
windows_executable	advanced_windows_executable, CMS_windows_executable
X11_lua	x11_flex
xor_executable_lua	xor_executable

[RSAAdmin]

This is great information - as there is no Matrix that I have found and simply configuring a new LUA parser does not automatically deprecate the equivalent flex parser.  Please keep this list updated as LUA parsers continue.  Also - some questions regarding your list as I have gone through this exercise:

bittorrent_lua - doesn't exist

botnet_lua - doesn't exist

DNS_verbose_lua - doesn't exist

fingerprint_pdf_lua - does this also replace malware_pdf_v201?

fingerprint_php_lua - doesn't exist

fingerprint_key - doesn't exist

htran_lua - doesn't exist

IRC_verbose_lua - does this also cover irc-expanded?

As of this writing I was not able to search the above lua parsers (those indicated as doesn't exist). 

[RSAAdmin]

Not all of the listed lua parsers are in Live yet, but will be soon.  Several were posted to Live on Friday.

For fingerprint_pdf_lua:  yes (malware_pdf and malware_pdf_v201 are the same thing)

For IRC_verbose_lua: yes - I missed including irc-expanded in the list

That should cover nearly all of the existing flex parsers in Live, except for some website-specific parsers that will likely be converted in a more generic form, and some that won't be converted at all (e.g., "tcp-flags").  Many of the native parsers will eventually have lua equivalents, but those will be slower in coming.

[RSAAdmin]

Another I missed:

mail_lua also replaces email-ip

[Anonymous]

Can the source of the lua parsers be viewed directly?

It would be very helpful to reference these when creating my own parsers.

[RSAAdmin]

Currently they are all encrypted.

There will be some made available in some manner as unencrypted for demonstration purposes. But I don't yet know which, when, or how.

[Anonymous]

Thanks for your response.

[Anonymous]

Now that HTTP_lua is available and replaces NTLMSSP does it also replace NTLMSSP_lua?