Background
Investigations involve hypothesis generation and hypothesis evaluation to assess the strength of evidence in light of competing hypotheses.
Requirements
Requirement 1
State a Hypothesis and one or more alternative Hypotheses being considered. Selecting a single hypothesis without consideration of alternatives increases risk of confirmation bias.
Requirement 2
Evaluate evidence under multiple hypotheses and assign an evidenceEvaluation value to the observations, not the hypothesis.
Requirement 3
Allow for different types of evidenceEvaluation, including verbal (more probable), numeric (0.9), pseudonumeric (C5.5)
Requirement 4
Represent who performed the evaluation, using which tools and methods.
Requirement 5
Group a set of hypotheses being considered
Risk / Benefit analysis
Benefits
The ability to generate and evaluate hypotheses is fundamental to cyber-investigations.
Risks
The analysis of competing hypotheses approach represented here are incompatible with conclusion scales (e.g., Identification, Inconclusive, and Exclusion), which typically only consider one hypothesis (problematic because of confirmation bias), are categorical (problematic because of lack of agreement and cliff-edge effect), and do not make a clear distinction between hypotheses evaluation (prosector's fallacy) and evidence evaluation.
The evidence evaluation approach represented here is incompatible with non-scientific domains (e.g., intelligence analysis) approaches that assign a likelihood of occurrence of an event or development, and the confidence in the basis for this judgment. Assigning a value to an analytic judgment evaluates the hypotheses not the evidence (prosector's fallacy), increases risk of confirmation bias, and is not balanced with evaluation of opposing hypothesis.
Competencies demonstrated
Competency 1
Evaluate the evidence given opposing hypotheses.
Competency Question 1.1
Are the [Observations] more probable given [Hypothesis 1] or [Hypothesis 2]?
Result 1.1
The [Observations] are more probable given [Hypothesis 1] rather than [Hypothesis 2].
Competency Question 1.2
What [Observations] were evaluated under [Hypothesis 1] and [Hypothesis 2]?
Result 1.2
The [Observations] were evaluated under [Hypothesis 1] and [Hypothesis 2].
Competency 2
Track provenance and ensure accountability for evaluation of evidence given opposing hypotheses.
Competency Question 2.1
Who performed the evaluation of the evidence given opposing hypotheses?
Result 2.1
Eoghan Casey performed the evaluation of the evidence given opposing hypotheses.
Solution suggestion
Add Hypothesis (Assertion): A statement that is either true or false, the truth of which is uncertain
Add EvidenceEvaluationResult (AnalyticResult): An opinion that is formed on the basis of observed objects (evidence) in light of a given hypothesis.
Add HypothesisGroup (Collection): A collection of hypotheses being considered for evidence evaluation.
This illustrative example demonstrates Requirement 1:
https://github.com/casework/CASE-Examples/tree/ONT-434/examples/illustrations/inference
Background
Investigations involve hypothesis generation and hypothesis evaluation to assess the strength of evidence in light of competing hypotheses.
Requirements
Requirement 1
State a Hypothesis and one or more alternative Hypotheses being considered. Selecting a single hypothesis without consideration of alternatives increases risk of confirmation bias.
Requirement 2
Evaluate evidence under multiple hypotheses and assign an evidenceEvaluation value to the observations, not the hypothesis.
Requirement 3
Allow for different types of evidenceEvaluation, including verbal (more probable), numeric (0.9), pseudonumeric (C5.5)
Requirement 4
Represent who performed the evaluation, using which tools and methods.
Requirement 5
Group a set of hypotheses being considered
Risk / Benefit analysis
Benefits
The ability to generate and evaluate hypotheses is fundamental to cyber-investigations.
Risks
The analysis of competing hypotheses approach represented here are incompatible with conclusion scales (e.g., Identification, Inconclusive, and Exclusion), which typically only consider one hypothesis (problematic because of confirmation bias), are categorical (problematic because of lack of agreement and cliff-edge effect), and do not make a clear distinction between hypotheses evaluation (prosector's fallacy) and evidence evaluation.
The evidence evaluation approach represented here is incompatible with non-scientific domains (e.g., intelligence analysis) approaches that assign a likelihood of occurrence of an event or development, and the confidence in the basis for this judgment. Assigning a value to an analytic judgment evaluates the hypotheses not the evidence (prosector's fallacy), increases risk of confirmation bias, and is not balanced with evaluation of opposing hypothesis.
Competencies demonstrated
Competency 1
Evaluate the evidence given opposing hypotheses.
Competency Question 1.1
Are the [Observations] more probable given [Hypothesis 1] or [Hypothesis 2]?
Result 1.1
The [Observations] are more probable given [Hypothesis 1] rather than [Hypothesis 2].
Competency Question 1.2
What [Observations] were evaluated under [Hypothesis 1] and [Hypothesis 2]?
Result 1.2
The [Observations] were evaluated under [Hypothesis 1] and [Hypothesis 2].
Competency 2
Track provenance and ensure accountability for evaluation of evidence given opposing hypotheses.
Competency Question 2.1
Who performed the evaluation of the evidence given opposing hypotheses?
Result 2.1
Eoghan Casey performed the evaluation of the evidence given opposing hypotheses.
Solution suggestion
Add Hypothesis (Assertion): A statement that is either true or false, the truth of which is uncertain
Add EvidenceEvaluationResult (AnalyticResult): An opinion that is formed on the basis of observed objects (evidence) in light of a given hypothesis.
Add HypothesisGroup (Collection): A collection of hypotheses being considered for evidence evaluation.
This illustrative example demonstrates Requirement 1:
https://github.com/casework/CASE-Examples/tree/ONT-434/examples/illustrations/inference