Email log parsing - subject encoded (RFC 2047)

[JohnDoe1]

Hello,

I have a cisco ironport esa wich provide the following kind of log:

Jun 06 14:00:07 mail_logs4: Info: MID 62990685 Subject '=?ISO-8859-1?B?SWYgeW91IGNhbiByZWFkIHRoaXMgeW8=?=\r\n\t=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=\r\n\t=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=\r\n\t=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?='

Does anyone have developed a (lua) parser to decode the encoded subject to human readable format?

(online decoder Online MIME Headers Decoder (RFC 2047))

[DavidWaugh1]

Hi that is an interesting issue. Can you give me a full line of a log so that I can test it in my lab please. No promises but lets see...

[JohnDoe1]

Thanks David Waugh​!

It's already the full line (datetime / mailid / subject):

Jun 06 14:00:07 mail_logs4: Info: MID 62990685 Subject '=?ISO-8859-1?B?SWYgeW91IGNhbiByZWFkIHRoaXMgeW8=?=\r\n\t=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=\r\n\t=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=\r\n\t=?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?='

This issue/feature has recently been reported to  Cisco Bug: CSCun16129 - Translate subjects from base64 to human readable format for export

Moreover:

• Sometimes, there is no '\r\n\t' but just '\r\n '
• Subject is not always encoded
• Encoding varies (utf-8, ISO, etc)

[DavidWaugh1]

Great. making some progress here.

I use:

Lua: demo

And have the following code as a basis:

Paste in the top box:

function mimedecode(mystring)
t={}
for k,v in string.gmatch(mystring,"=%?ISO%-8859%-%d%?B%?(.-)%?=") do
 t =v
 print("Variable")
 print(dec(k))
end
end
-- Lua 5.1+ base64 v3.0 (c) 2009 by Alex Kloss <alexthkloss@web.de>
-- licensed under the terms of the LGPL2
-- character table string
local b='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
-- encoding
function enc(data)
    return ((data:gsub('.', function(x) 
        local r,b='',x:byte()
        for i=8,1,-1 do r=r..(b%2^i-b%2^(i-1)>0 and '1' or '0') end
        return r;
    end)..'0000'):gsub('%d%d%d?%d?%d?%d?', function(x)
        if (#x < 6) then return '' end
        local c=0
        for i=1,6 do c=c+(x:sub(i,i)=='1' and 2^(6-i) or 0) end
        return b:sub(c+1,c+1)
    end)..({ '', '==', '=' })[#data%3+1])
end
-- decoding
function dec(data)
    data = string.gsub(data, '[^'..b..'=]', '')
    return (data:gsub('.', function(x)
        if (x == '=') then return '' end
        local r,f='',(b:find(x)-1)
        for i=6,1,-1 do r=r..(f%2^i-f%2^(i-1)>0 and '1' or '0') end
        return r;
    end):gsub('%d%d%d?%d?%d?%d?%d?%d?', function(x)
        if (#x ~= 8) then return '' end
        local c=0
        for i=1,8 do c=c+(x:sub(i,i)=='1' and 2^(8-i) or 0) end
        return string.char(c)
    end))
end
mimedecode("=?ISO-8859-1?B?SWYgeW91IGNhbiByZWFkIHRoaXMgeW8=?= =?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=")

Now click on run and you get the results:

Variable

If you can read this yo

Variable

u understand the example.

Its unfinished but getting somewhere....

[DavidWaugh1 · Accepted solution]

Okay I have a working one!

Here is the code attached below.

[JohnDoe1]

Thanks David Waugh​

I just check the script parserbook/example parsers/mal.lua  and I found the  function:

function mailParser:rfc2047(encodedString)
    if not encodedString then
        return
    end
    local decodedString, charset = encodedString, nil
    -- look for encoding, e.g.:
    --     =?windows-1256?B?VklQIFRv...=?
    if string.byte(encodedString, 1) == 61 and string.byte(encodedString, 2) == 63 then
        local charsetEnd = string.find(encodedString, "?", 3)
        if charsetEnd then
            if string.byte(encodedString, charsetEnd + 2) == 63 then
                local encodingChar = string.byte(encodedString, charsetEnd + 1)
                if encodingChar == 66 or encodingChar == 81 or encodingChar == 98 or encodingChar == 113 then
                    local stringBegin = charsetEnd + 3
                    local stringEnd = string.find(encodedString, "?=", stringBegin + 1)
                    if stringEnd then
                        stringEnd = stringEnd - 1
                        decodedString = string.sub(encodedString, stringBegin, stringEnd)
                        decodedString = string.gsub(decodedString, "_", "\032")
                        if encodingChar == 66 or encodingChar == 98 then
                            decodedString = nw.base64Decode(decodedString)
                        else
                            decodedString = nwll.decodeQuotedPrintable(decodedString)
                        end
                        charset = string.sub(encodedString, 3, charsetEnd - 1)
                    end
                end
            end
        end
    end
    return decodedString, charset
end

I have to check/adapt the script

[WilliamMotley1]

Actually, MAIL_lua has been substantially rewritten from what's in the book and I haven't found the time to update the book yet.  Here's how it looks now:

local function rfc2047(encodedString)
    if not encodedString then
        return
    end
    local decodedString, charset, encoding
    -- look for encoding, e.g.,  =?windows-1256?B?VklQIFRv...?=
    charset, encoding, decodedString = string.match(encodedString, "^%s-=%?([^?]+)%?([BbQq])%?([^?]+)%?=")
    if charset and encoding and decodedString then
        charset = string.match(charset, "^([^*]+)%*") or charset
        decodedString = string.gsub(decodedString, "_", " ")
        if encoding == "B" or encoding == "b" then
            decodedString = nw.base64Decode(decodedString) or decodedString
        elseif encoding == "Q" or encoding == "q" then
            decodedString = nwll.decodeQuotedPrintable(decodedString) or decodedString
        end
    end
    decodedString = decodedString or encodedString
    return decodedString, charset
end

[JohnDoe1]

It works! Thanks David Waugh​ and William Motley​