Sensitivity Translation Bug

[smsullivan]

Issue discovered when translating the OUO field from a CFM13 file. Translation is from cfm13 to lqmtools schema. The OUO field maps to the sensitivity field in the lqmtools schema.

The CFM13 file in question has multiple indicators in said file. When there are multiple indicators with OUO values set to the same value (ex: OUO = 1 or OUO = 0) there is no issue, but the moment the chain is broken all of the resulting sensitivity fields are marked as noSensitivity in the resulting translations. Example below

Working block in short-hand:
additional data -- type="integer" -- meaning="OUO" value=1
...
additional data -- type="integer" -- meaning="OUO" value=1
...
additional data -- type="integer" -- meaning="OUO" value=1
result:
action1, sensitivity
block, OUO
block, OUO
block, OUO

Broken block in short-hand:
additional data -- type="integer" -- meaning="OUO" value=1
...
additional data -- type="integer" -- meaning="OUO" value=0
...
additional data -- type="integer" -- meaning="OUO" value=1
result:
action1, sensitivity
block, noSensitivity
block, noSensitivity
block, noSensitivity
expected result:
action1, senstivity
block, OUO
block, noSensitivity
block, OUO

[mghenderson]

Just ran a test against this with an input of cfm13alert (modified from test case) and output of lqmt, and I'm getting expected values. Starting with an input w/ 3 alerts, and sensitivity of ouo/noSensitivity/ouo, I achieve the expected results

Note: In my testing, I found if you're ingesting a metadata file along with the CFM13Alert, it should override all values to the value found in the CFM13Alert file.

Input File:

<!DOCTYPE IDMEF-Message PUBLIC "-//IETF//DTD RFC XXXX IDMEF v1.0//EN" "idmef-message.dtd">
<IDMEF-Message xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.anl.gov/cfm/1.3/IDMEF-Message" xsi:schemaLocation="http://www.anl.gov/cfm/1.3/IDMEF-Message/../../../resources/schemas/CFMMessage13.xsd">
    <Alert>
        <Analyzer analyzerid="Fake">
            <Node>
                <location>Fake National Lab</location>
                <name>Fake Name</name>
            </Node>
        </Analyzer>
        <AnalyzerTime>2016-02-21T22:50:02+0600</AnalyzerTime>
        <AdditionalData meaning="report schedule" type="string">5 minutes</AdditionalData>
        <AdditionalData meaning="number of alerts in this report" type="integer">2</AdditionalData>
        <AdditionalData meaning="report type" type="string">alerts</AdditionalData>
        <AdditionalData meaning="report start time" type="date-time">2016-02-21T22:45:53+0700</AdditionalData>
    </Alert>
    <Alert>
        <CreateTime>2016-02-21T22:45:53-0400</CreateTime>
        <Source>
            <Node>
                <Address category="ipv4-addr">
                     <address>5.5.5.5</address>
                </Address>
            </Node>
        </Source>
        <Target>
            <Service>
                <port>22</port>
                <protocol>TCP</protocol>
            </Service>
        </Target>
        <Classification text="SSH scans against multiple hosts, direction:ingress, confidence:87, severity:high">
            <Reference meaning="Scanning" origin="user-specific">
                <name>SSH Attack</name>
                <url> </url>
            </Reference>
        </Classification>
        <Assessment>
            <Action category="block-installed"/>
        </Assessment>
        <AdditionalData meaning="restriction" type="string">public</AdditionalData>
        <AdditionalData meaning="recon" type="integer">1</AdditionalData>
        <AdditionalData meaning="prior offenses" type="integer">11</AdditionalData>
        <AdditionalData meaning="duration" type="integer">86400</AdditionalData>
        <AdditionalData meaning="alert threshold" type="integer">0</AdditionalData>
        <AdditionalData meaning="OUO" type="integer">1</AdditionalData>
        <AdditionalData meaning="top level domain owner" type="string">The Republic of Fake</AdditionalData>
    </Alert>
    <Alert>
        <CreateTime>2016-02-21T22:45:53-0400</CreateTime>
        <Source>
            <Node>
                <Address category="ipv4-addr">
                    <address>10.10.10.10</address>
                </Address>
            </Node>
        </Source>
        <Target>
            <Service>
                <port>22</port>
                <protocol>TCP</protocol>
            </Service>
        </Target>
        <Classification text="SSH scans against multiple hosts, direction:ingress, confidence:87, severity:high">
            <Reference meaning="Scanning" origin="user-specific">
                <name>SSH Attack</name>
                <url> </url>
            </Reference>
        </Classification>
        <Assessment>
            <Action category="block-installed"/>
        </Assessment>
        <AdditionalData meaning="restriction" type="string">private</AdditionalData>
        <AdditionalData meaning="recon" type="integer">0</AdditionalData>
        <AdditionalData meaning="prior offenses" type="integer">11</AdditionalData>
        <AdditionalData meaning="duration" type="integer">86400</AdditionalData>
        <AdditionalData meaning="alert threshold" type="integer">0</AdditionalData>
        <AdditionalData meaning="OUO" type="integer">0</AdditionalData>
        <AdditionalData meaning="top level domain owner" type="string">The Republic of Fake</AdditionalData>
    </Alert>
    <Alert>
        <CreateTime>2016-02-21T22:45:53-0400</CreateTime>
        <Source>
            <Node>
                <Address category="ipv4-addr">
                    <address>20.20.20.20</address>
                </Address>
            </Node>
        </Source>
        <Target>
            <Service>
                <port>22</port>
                <protocol>TCP</protocol>
            </Service>
        </Target>
        <Classification text="SSH scans against multiple hosts, direction:ingress, confidence:87, severity:high">
            <Reference meaning="Scanning" origin="user-specific">
                 <name>SSH Attack</name>
                 <url> </url>
            </Reference>
        </Classification>
        <Assessment>
            <Action category="block-installed"/>
        </Assessment>
        <AdditionalData meaning="restriction" type="string">private</AdditionalData>
        <AdditionalData meaning="recon" type="integer">0</AdditionalData>
        <AdditionalData meaning="prior offenses" type="integer">11</AdditionalData>
        <AdditionalData meaning="duration" type="integer">86400</AdditionalData>
        <AdditionalData meaning="alert threshold" type="integer">0</AdditionalData>
        <AdditionalData meaning="OUO" type="integer">1</AdditionalData>
        <AdditionalData meaning="top level domain owner" type="string">The Republic of Fake</AdditionalData>
    </Alert>
</IDMEF-Message>

Output:

[
    {
        "action1": "Block",
        "comment": "SSH scans against multiple hosts, direction:ingress, confidence:87, severity:high",
        "detectedTime": 1456116353.0,
        "directSource": "Fake",
        "duration1": "86400",
        "fileHasMore": "0",
        "indicator": "5.5.5.5",
        "indicatorType": "IPv4Address",
        "majorTags": "Scanning",
        "priors": "11",
        "processedTime": "1467671787",
        "reason1": "SSH Attack",
        "reconAllowed": "0",
        "reference1": "user-specific",
        "restriction": "WHITE",
        "sensitivity": "ouo"
    },
    {
        "action1": "Block",
        "comment": "SSH scans against multiple hosts, direction:ingress, confidence:87, severity:high",
        "detectedTime": 1456116353.0,
        "directSource": "Fake",
        "duration1": "86400",
        "fileHasMore": "0",
        "indicator": "10.10.10.10",
        "indicatorType": "IPv4Address",
        "majorTags": "Scanning",
        "priors": "11",
        "processedTime": "1467671787",
        "reason1": "SSH Attack",
        "reconAllowed": "1",
        "reference1": "user-specific",
        "restriction": "AMBER",
        "sensitivity": "noSensitivity"
    },
    {
        "action1": "Block",
        "comment": "SSH scans against multiple hosts, direction:ingress, confidence:87, severity:high",
        "detectedTime": 1456116353.0,
        "directSource": "Fake",
        "duration1": "86400",
        "fileHasMore": "0",
        "indicator": "20.20.20.20",
        "indicatorType": "IPv4Address",
        "majorTags": "Scanning",
        "priors": "11",
        "processedTime": "1467671787",
        "reason1": "SSH Attack",
        "reconAllowed": "1",
        "reference1": "user-specific",
        "restriction": "AMBER",
        "sensitivity": "ouo"
    }
]

[dharknessANL]

The metadata file (envelope) should only be used to set values (sensitivity, recon, etc.) when not present in an alert. In other words, the precedence order should be (highest to lowest): internal, metadata (envelope), defaults.

[smsullivan]

Appreciate the clarification Dan. Just confirmed all the alerts I was running to get this bug were using Metadata files that specified the sensitivity.

[mghenderson]

I'll look into changing how we handle metadata files. I'm thinking the original use-case for them was pretty limited, so I'll want to review & go over new use-cases that before changing anything.