tests: end-to-end detection coverage (engine C2/ICS + APT/MITRE plugins) - #7
Open
eris-ot wants to merge 2 commits into
Open
tests: end-to-end detection coverage (engine C2/ICS + APT/MITRE plugins)#7eris-ot wants to merge 2 commits into
eris-ot wants to merge 2 commits into
Conversation
added 2 commits
May 18, 2026 04:08
The engine's detection logic — its headline capability — and the
report-facing plugins had no behavioral test (existing files only fed
hand-built report dicts to downstream consumers).
scapy generates 9 deterministic scenario PCAPs (benign OT/IT baseline +
one injected pattern each, tuned to thresholds read from engine.py),
the real engine runs on each, and findings are asserted:
engine risk_findings / c2_indicators:
C2_BEACONING, C2_PERSISTENCE, C2_DNS_EXFIL, C2_SUSPECT_CHANNEL,
C2_DATA_EXFIL, ICS_EXTERNAL_COMMS, PORT_SCAN_TARGET,
MODBUS_WRITE_ANON (+ clean negative control / no false positives)
plugins (end-to-end through the report):
marlinspike-apt -> APT_LATERAL_MOVEMENT_SMB attribution
marlinspike-mitre -> C2_BEACONING classified to ATT&CK T1071
Stage 4b malware -> integration test gated by binary presence
(skips cleanly; the Rust engine is Docker-only)
Tests skip when scapy or tshark/DPI is unavailable, so a bare CI image
stays green. Generated PCAPs are gitignored (regenerated to tmp).
Suite: 380 passed, 1 skipped; ruff clean.
Compiled the marlinspike-malware Rust engine locally; the rules repo ships a deterministic self-validation IOC (bootstrap-bad-host: dns_query == bad.example.invalid). New malware_ioc fixture issues that DNS lookup so Stage 4b produces a genuine end-to-end signature hit, asserted to merge into both risk_findings and c2_indicators as MALWARE_IOC_MATCH (plus a clean negative). Still binary-gated: skips cleanly where the Rust engine is absent (bare CI), runs for real wherever it exists (local / Docker / CI-with-binary). Suite: 382 passed (10 new detection assertions), ruff clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
End-to-end detection coverage
The engine's detection logic — the product's headline capability — and the report-facing plugins had no behavioral test. The ~10 existing files that reference
risk_findings/C2 only feed hand-built report dicts into downstream consumers (emitters, aggregation); none ever ran the detectors against packets. A regression in the beacon scorer or a moved entropy threshold would have shipped silently.This adds a scapy fixture generator + two test modules. Each scenario = a shared benign OT/IT baseline plus one injected attack pattern, tuned to thresholds read directly out of
engine.py; the real engine runs on every PCAP and its own output is asserted.Engine detectors pinned (
test_detection_scenarios.py)cleanc2_beaconC2_BEACONINGCRIT +C2_PERSISTENCEdns_exfilC2_DNS_EXFILCRITc2_suspect_channelC2_SUSPECT_CHANNELHIGHc2_data_exfilC2_DATA_EXFILHIGHics_externalICS_EXTERNAL_COMMSCRITport_scanPORT_SCAN_TARGETHIGHmodbus_writeMODBUS_WRITE_ANONMEDlateral_smb5 of the 7 distinct C2 indicator types are pinned. The other two (
C2_DNS_HIGH_ENTROPY,C2_DNS_TUNNEL_SUSPECT) are lower DNS-tunnel tiers thedns_exfilpattern deliberately escalates past to the topC2_DNS_EXFILtier — pinning them would mean asserting a weaker classification of the same traffic; called out rather than padded.Plugins end-to-end (
test_detection_plugins.py)marlinspike-apt:lateral_smb→APT_LATERAL_MOVEMENT_SMBattributed to10.10.0.40(+ clean negative).marlinspike-mitre:C2_BEACONING→ ATT&CK T1071 on anobservedbasis (this plugin had no test at all).marlinspike-malwareRust engine exists (Docker/CI-with-binary), skips cleanly here. It asserts the integration invariant, not a synthetic signature hit (rule packs are external/uncontrolled — stated honestly in the test).Safety
Tests
skipwhenscapyortshark/DPI is absent, so a bare CI image stays green. Generated PCAPs are gitignored (regenerated to a tmp dir per run). Suite: 380 passed, 1 skipped; ruff clean.