NULLFIRE

Detection Matchability Analyzer

Find the detections that can never see your data.

Zero alerts means one of two things

Your SIEM reports no alerts for T1059.001 this quarter.

┌─────────────────────────────┐        ┌─────────────────────────────┐
│  1.  Nobody abused          │   or   │  2.  The rule cannot fire.  │
│      PowerShell.            │        │                             │
│                             │        │  It references a field your │
│      Good news.             │        │  pipeline renamed two years │
│      Do nothing.            │        │  ago. It has been dead ever │
│                             │        │  since. Nobody noticed.     │
└─────────────────────────────┘        └─────────────────────────────┘

Identical from a dashboard. Opposite responses.

Detection rules rot quietly. A rule can reference a field the ingestion pipeline renamed, a process name nobody runs any more, a Sysmon field that stopped arriving when you moved to Windows Event Forwarding, or an event schema that changed two vendor versions ago. Nothing breaks loudly. The rule stays in the repository, keeps its ATT&CK tags, keeps appearing in coverage dashboards, keeps counting as audit evidence — and never fires again.

Nullfire compares your Sigma rules against a sample of the logs you actually have, after your normalization pipeline, and tells you which rules still have a viable matching path.

60-second start

pip install -e .

# 1. What is actually in your data? (start here, always)
nullfire profile --logs events.jsonl

# 2. Which rules can fire against it?
nullfire analyze --rules rules/ --logs events.jsonl \
    --pipeline pipeline.yml \
    --declare-source "product=windows,category=process_creation"

Or run the bundled demo, which needs no data of your own:

nullfire analyze --config demo/config/nullfire.yml

See what that prints

NULLFIRE
Detection Matchability Analyzer

Rules analyzed              20
Logs analyzed            1,200
Pipeline            nullfire-demo-ecs-windows (file: ecs-windows.yml)
Declared sources    product=windows, category=process_creation; ...

MATCHABLE                    7
DEGRADED                     5
NULLFIRE                     4
UNASSESSED                   2   (not a finding)
ERRORS                       2   (operational, not detection findings)

ATT&CK impact
-------------
T1059.001    1 of 3 rule(s) cannot match  [execution]
T1003.001    1 of 2 rule(s) cannot match  [credential-access]
T1078.003    1 of 1 rule(s) cannot match  [persistence]
T1548.002    1 of 1 rule(s) cannot match  [privilege-escalation]

4 rule(s) across 4 ATT&CK technique(s) currently contribute claimed rule
coverage but cannot match the supplied data. Coverage dashboards counting
those rules are overstating what is able to fire.

NULLFIRE rules
--------------

  Office Application Spawning Script Interpreter
    id        4f8b1a20-0006-4c6a-9c11-a1b2c3d40006
    path      windows/nullfire_parent_command_line.yml
    attack    T1059.001
    reason    The rule requires 'process.parent.command_line', which does not
              appear in any of the 1,200 supplied events. The dataset is
              declared to contain this rule's log source, so the field is
              absent rather than merely unrepresented: no event produced by
              this pipeline can satisfy the rule as written.
    missing   process.parent.command_line
    fixture   minimal event that would satisfy this rule:
                {
                  "event.code": 4688,
                  "process.executable": "\\cmd.exe",
                  "process.parent.command_line": "OUTLOOK.EXE"
                }

CI policy
---------
DEGRADED <= 20              PASS
NULLFIRE <= 0               FAIL
mode                        fail (violations cause a non-zero exit)

RESULT: FAIL

What it tells you

	Status	Meaning
🟢	MATCHABLE	Every field the rule needs exists, and each decidable predicate had a matching value observed. There is a viable matching path. Not a claim that the detection is good.
🟡	DEGRADED	A path may exist, but the evidence is weak — a literal was never observed, a field is sparse, only some branches resolve, or part of the logic is undecidable.
🔴	NULLFIRE	A required dependency cannot be satisfied against the observed schema. No event this pipeline produces can satisfy this rule.
⚪	UNASSESSED	The rule's log source is not represented in your sample. This is not a finding. Nullfire had nothing to judge it against.

Operational failures get their own bucket and never wear a detection status — PARSE_ERROR, PIPELINE_ERROR, VERIFICATION_ERROR, CONFIG_ERROR. A rule that fails to parse has no verdict at all. That separation is enforced by an assertion in the data model, not by convention.

DEGRADED vs NULLFIRE, concretely

Rule requires:   process.executable  endswith  '\powershell.exe'

Observed:        process.executable   present in 67.5% of events
                 powershell.exe       observed 0 times
                 pwsh.exe             observed 116 times

Verdict:         🟡 DEGRADED
                 The field is healthy. This environment moved to PowerShell 7,
                 so the literal has never appeared — but a sample cannot prove
                 it never will. "Not observed in supplied data" means exactly
                 that, and nothing more.

What it is not

Being precise about this matters more than the feature list.

Important

Nullfire does not prove a detection is correct. A rule can be MATCHABLE and still be badly written, mis-scoped or trivially evadable.
Nullfire does not prove an attack would be detected in production. It reads a sample, not your live pipeline, and reasons about schema and observed values — not about adversary behaviour.
Nullfire does not replace detection testing or adversary simulation. It tells you which rules are worth testing.

It is a rule/data compatibility validator. Narrow job, done carefully.

How it works

flowchart TD
    A["nullfire.yml<br/>config"] -->|"strict validation<br/>unknown keys are fatal"| P
    B["pipeline.yml<br/>Sigma processing"] --> P["pipeline.py"]
    P -->|"FAILURE ABORTS THE RUN"| STOP(["PIPELINE_ERROR<br/>exit 2 · nothing judged"])
    C["events.jsonl<br/>post-pipeline logs"] --> PR["profile.py<br/>stream · flatten · bound"]
    D["rules/<br/>Sigma corpus"] --> R["rules.py<br/>one file at a time"]
    R -->|"apply pipeline<br/>→ effective rule"| K["constraints.py<br/>AST → constraint tree"]
    K --> S["satisfy.py<br/>four-valued lattice"]
    K --> F["fixture.py<br/>minimal event<br/>+ concrete verification"]
    PR --> S
    S --> V["verdict.py<br/>gating + interlocks"]
    V --> OUT["report.py<br/>text · JSON"]
    F --> OUT
    OUT --> CLI(["cli.py<br/>exit 0/1/2/3/4"])

    style STOP fill:#4a1010,stroke:#c04040,color:#fff
    style S fill:#10304a,stroke:#4090c0,color:#fff
    style V fill:#10304a,stroke:#4090c0,color:#fff
    style F fill:#103a20,stroke:#40a060,color:#fff

The four-valued lattice — why three values are not enough

Every leaf predicate evaluates to one of four values, ordered:

UNSAT   <   UNKNOWN   <   WEAK   <   SAT

Value	Meaning
`SAT`	Field exists and a satisfying value was observed
`WEAK`	Field exists, but the required value was never observed, or the field is too sparse to rely on
`UNKNOWN`	Undecidable statically — a regex, a `fieldref`, or a value set we stopped collecting
`UNSAT`	The field appears nowhere in the observed schema

Then the whole engine is two lines:

AND = min          OR = max

"I proved this cannot match", "I could not decide", and "this could match but I saw no evidence" are three genuinely different states. Collapsing any two produces either false confidence or useless output.

The useful properties fall out rather than being special-cased:

An absent optional OR branch cannot sink a healthy rule — max(UNSAT, SAT) = SAT
One unsatisfiable conjunct is enough — min(SAT, SAT, UNSAT) = UNSAT
Uncertainty propagates instead of being rounded away — min(SAT, UNKNOWN) = UNKNOWN

NOT is asymmetric, deliberately. Sigma NOT branches are filters — they subtract matches, they essentially never make a rule impossible. So negation never returns UNSAT, except via one narrow provable check: a conjunction that requires F == V and simultaneously excludes exactly F == V.

Declaring a rule dead because of a filter clause would be the worst false positive this tool could produce. Filters are the most-edited part of a rule, and an engineer told their rule was dead because of an exclusion would rightly stop trusting the output.

Full detail: docs/verdicts.md

How a NULLFIRE verdict is earned — two interlocks, both fail safe

NULLFIRE is the verdict that makes somebody open a ticket at 2am, so it sits behind two independent gates. Both only ever downgrade.

1 · You must declare what your data is.

A rule requiring EventID judged against data containing only event.code looks exactly like a rule written for the wrong log source. Both present as total field absence. Only you can break that tie:

data_sources:
  - product: windows
    category: process_creation
  - product: windows
    service: security

Without a declaration, the worst available verdict is DEGRADED, and every downgraded rule carries a note saying so.

An omitted dimension means unspecified, not wildcard — declaring only product: windows will not license NULLFIRE for every Windows rule regardless of category.

2 · The sample must be big enough.

Below min_sample_size (default 50) events, verdicts cap at DEGRADED. You cannot conclude a rule is dead from a three-line log file.

The one exception: a rule that contradicts itself — requiring and excluding the same literal — is reported NULLFIRE regardless. That conclusion is a property of the rule and does not depend on your data at all.

Field mapping — why raw rules are never compared to logs

raw rule  →  Sigma processing pipeline  →  effective rule  →  data comparison

A rule saying EventID: 4688 and an event saying event.code: "4688" are compatible if your pipeline maps them. Nullfire loads a real pySigma processing pipeline and reports verdicts against the effective rule — the one actually compared against data.

nullfire analyze --rules rules/ --logs events.jsonl \
    --pipeline demo/pipeline/ecs-windows.yml

Pipelines are plain pySigma YAML — no plugin install, no network. An installed pySigma pipeline name works too.

⚠️ If a pipeline cannot be loaded, Nullfire analyses nothing. It reports a PIPELINE_ERROR and exits 2. A typo'd pipeline name that instead reported three thousand rules as dead would look exactly like a catastrophic outage, and that is the single worst output this tool could produce.

Minimal fixtures — turn "this rule is dead" into a regression test

For NULLFIRE and DEGRADED rules, Nullfire generates the smallest synthetic event that satisfies the detection logic.

{
  "rule_id": "4f8b1a20-0006-4c6a-9c11-a1b2c3d40006",
  "status": "NULLFIRE",
  "fixture": {
    "event.code": 4688,
    "process.executable": "\\cmd.exe",
    "process.parent.command_line": "OUTLOOK.EXE"
  }
}

Three fields, nothing else. Drop it into a detection test and the rule fails until the underlying gap is fixed.

Two guarantees:

Nothing is emitted unverified. Every candidate is evaluated against the rule's own constraint tree with real boolean semantics — including true negation — before it is offered. This is a separate evaluator from the lattice one, precisely because the lattice treats every NOT as satisfiable.
Refusal is a valid answer. Regex, fieldref, unresolved placeholders and irreconcilable field requirements each produce an explicit "not generated, because…" rather than a plausible-looking guess.

Fixtures are generated from the rule alone, never the data — so a fixture cannot leak log content, which is also why they survive --redact.

Values are either literals the rule itself requires or the inert placeholder NULLFIRE_TEST. No credentials, tokens, payloads or destructive commands are synthesised, and nothing is ever executed.

nullfire analyze --config nullfire.yml --fixtures-out out/fixtures.jsonl

ATT&CK coverage impact — which techniques are propped up by dead rules

T1059.001    1 of 3 rule(s) cannot match  [execution]
T1003.001    1 of 2 rule(s) cannot match  [credential-access]

This is rule-inventory coverage, not detection capability. The question it answers is: which techniques are being falsely represented by dead rules?

📌 A technique with ten MATCHABLE rules is not "detected" — those rules may be badly written. A technique with three NULLFIRE rules is not "undetected" — something else may cover it. What the rollup does tell you is where a coverage dashboard is counting rules that cannot fire.

UNASSESSED rules are reported separately and never counted as failures. A technique whose rules are all UNASSESSED means your sample was incomplete.

The disclaimer travels in the JSON output too, under attack.note — please carry it through if you render this anywhere.

The demo, and its point

nullfire analyze --config demo/config/nullfire.yml

Twenty hand-written rules over 1,200 synthetic ECS-shaped Windows events. Each rule file is named for the verdict it exists to provoke, and a test asserts all twenty against their filenames — so these numbers cannot drift out of step with the code.

Now run it again with the data-source declaration removed:

nullfire analyze --config demo/config/nullfire-undeclared.yml

Same rules. Same events. Same pipeline. The only difference is one config block:

	declared	undeclared
🟢 MATCHABLE	7	7
🟡 DEGRADED	5	7
🔴 NULLFIRE	4	1
⚪ UNASSESSED	2	3

Three of the four dead rules become DEGRADED or UNASSESSED, each carrying a note explaining that declaring the data source would sharpen the verdict.

That is the design, not a bug. A rule requiring a field that appears nowhere in your data is genuinely indistinguishable from a rule written for a different log source. Only you can break that tie. Until you do, Nullfire refuses to make the stronger claim.

The one NULLFIRE that survives is the self-contradictory rule — whose deadness is a property of the rule, not of your data.

→ demo/README.md walks through all twenty rules and why each gets its verdict.

Configuration

Full annotated config

rules_dir: ./rules
log_file: ./events.jsonl
pipeline: ./pipeline/ecs-windows.yml

# What this dataset actually IS. Declaring it is what licenses a NULLFIRE
# verdict. An omitted dimension means "unspecified", not "any".
data_sources:
  - product: windows
    category: process_creation

min_sample_size: 50          # below this, verdicts cap at DEGRADED
sparse_field_threshold: 0.02 # occupancy at or below which a field is sparse
max_distinct_values: 200     # per-field value-set cap
max_value_length: 4096       # per-value truncation
example_limit: 5             # example values shown per field
redact_values: false         # true withholds observed values from the report
generate_fixtures: true
max_events: 0                # 0 = no cap

ignored_rules: []            # by rule id or title
ignored_sources: []          # e.g. ["linux/auditd"]

policy:
  mode: fail                 # record | warn | fail
  nullfire:   { max: 0 }
  degraded:   { max: 20 }
  unassessed: { max: 50 }
  errors:     { max: 5 }

Validation is strict: unknown keys are fatal, and so are wrong types and out-of-range values. A gate configured with nullfir: {max: 0} that silently ignored the typo would report success forever. Every problem is reported at once, so fixing a config is one round trip.

An unconfigured threshold has no ceiling — None and 0 mean different things.

CI

- run: nullfire analyze --config nullfire.yml

Exit	Meaning
`0`	Completed; no enforced threshold exceeded
`1`	Completed; an enforced threshold was exceeded — the gate
`2`	Configuration or operational error: bad config, unloadable pipeline, empty log dataset
`3`	Rule processing failure — too many rules failed to parse for the result to mean anything
`4`	Unexpected internal error (a bug)

Tip

The split between 1 and 2 is the important one. A pipeline that treats "found a problem" and "could not run" identically eventually goes green because a path was mistyped, and nobody notices until it matters. An empty log file exits 2, not 0.

policy.mode decides whether a violation reaches the exit code at all: record counts it, warn reports it prominently, fail blocks the build.

Security

All input is untrusted — rule files, log files, pipelines and configuration alike. Rule repositories get cloned from the internet, and log files contain whatever an attacker managed to put into a log field.

	Property
⛔	No rule or log content is ever executed. No shell, no subprocess, no `eval`, no `exec`.
⛔	No regex from a rule file is ever compiled. Sigma wildcards are matched by a hand-written two-pointer matcher, O(n·m) worst case. Turning `abc` into `.a.b.c.` and handing it to a regex engine would reintroduce catastrophic backtracking reachable from a rule file. Sigma's `\|re` is reported undecidable instead.
🔒	YAML via `safe_load` only, in Nullfire and pySigma. Pipelines load with `allow_template_vars=False` and `allow_external_sources=False` passed explicitly, so a future default change cannot quietly widen this.
📏	Every input axis is bounded — streaming, per-line size, per-value length, distinct values per field, distinct field paths, nesting depth, tree depth and node count. And each bound is tracked, so hitting one weakens conclusions rather than silently changing them.
🚧	No path escapes the rules directory. Symlinks are not followed; candidates are fully resolved and containment-checked.
🕵️	Reports do not leak the host. Paths are relativized; a path outside the known roots degrades to its file name, never to an absolute path.
🏠	Logs stay local. No sockets, no telemetry, no upload. Malformed log lines are never quoted back in errors — that is exactly the content most likely to hold a secret. `--redact` withholds observed values entirely.

These are asserted by tests/test_security_properties.py, which walks the source AST — so a future change introducing exec, an unsafe YAML load, a socket, or a regex in the matching path fails the build rather than quietly falsifying this table.

→ docs/threat-model.md · SECURITY.md

Limitations

What Nullfire cannot tell you — read before acting on a report

Correlation rules are recorded but not analysed. Temporal windows, event counts and field grouping are out of scope, and guessing at them would produce confident nonsense.
Regex and fieldref are undecidable by design and reported as such rather than approximated.
NOT branches never yield NULLFIRE except via one narrow provable contradiction check — so Nullfire under-reports contradictions on purpose.
Verdicts describe the supplied sample, not production. A field absent from your sample may exist in your pipeline.
Keyword matching is approximated as containment. Real SIEM full-text indexing involves tokenisation Nullfire does not model, so keyword verdicts are the least precise it produces.
Fixture values follow the rule, not the data. A rule saying EventID: 4688 yields the integer 4688 even if your pipeline emits the string "4688".

Explicitly out of scope: live SIEM queries · attack simulation · EVTX parsing · automatic rule fixing · false-positive tuning · endpoint deployment · replacing a SIEM.

Full detail: docs/limitations.md

Docs


docs/verdicts.md	The lattice, the gates, and how each status is reached
docs/architecture.md	Module layout, data flow, design decisions
docs/json-schema.md	JSON report schema and how to consume it
docs/threat-model.md	Threats and mitigations
docs/limitations.md	What Nullfire cannot tell you
demo/README.md	All twenty demo rules, and why each gets its verdict

Install

python -m venv .venv
. .venv/Scripts/activate      # Windows
# source .venv/bin/activate   # Linux / macOS
pip install -e .

Python 3.10+. One direct runtime dependency: pySigma — Nullfire does not implement its own Sigma parser. The 3.10 floor comes from pySigma.

Commands

nullfire profile  --logs events.jsonl              # discover your real schema
nullfire analyze  --config nullfire.yml            # the main command
nullfire analyze  --config nullfire.yml --json     # machine-readable
nullfire validate --config nullfire.yml            # check a config, run nothing

Useful flags

Flag	Effect
`--declare-source "product=windows,category=process_creation"`	Declare the dataset inline; repeatable. This is what licenses NULLFIRE verdicts.
`--fail-on-nullfire N`	Fail the build above N dead rules (implies `mode: fail`)
`--fixtures-out PATH`	Write generated fixtures as a JSONL regression corpus
`--redact`	Withhold all observed log values — safe to attach to a ticket
`--show-profile`	Include the observed-data profile in the text report
`--detail-limit N`	Rules shown per detail section (`0` for summary only)
`--output PATH`	Write the report to a file

Try it on real rules

git clone --depth 1 https://github.com/SigmaHQ/sigma.git

nullfire analyze \
    --rules sigma/rules \
    --logs demo/logs/windows-ecs.jsonl \
    --pipeline demo/pipeline/ecs-windows.yml \
    --declare-source "product=windows,category=process_creation" \
    --declare-source "product=windows,service=security" \
    --declare-source "product=windows,service=system"

Expect a large UNASSESSED count — the demo dataset holds four event types and SigmaHQ covers hundreds. That is the correct answer, and it is why UNASSESSED exists as a status rather than being folded into a failure count.

SigmaHQ is licensed under the Detection Rule License and is not bundled here. Clone it yourself.

Development

pip install -e ".[dev]"
pytest                      # 424 tests
ruff check src tests
mypy

Validated against a real corpus: 3,144 SigmaHQ rules analysed in ~22 s on a laptop, with zero parse failures. Every NULLFIRE verdict in that run was audited to confirm it was driven by a genuinely absent field — no false positives, and no verdict without an explanation.

A detection rule that has never matched may be perfectly correct — or it may be impossible to match against your data pipeline.

Nullfire exists to tell you which.

MIT · Het Patel

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
demo		demo
docs		docs
src/nullfire		src/nullfire
tests		tests
.gitattributes		.gitattributes
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
pyproject.toml		pyproject.toml

Navigation Menu

Repository files navigation

NULLFIRE

Detection Matchability Analyzer

Zero alerts means one of two things

60-second start

What it tells you

DEGRADED vs NULLFIRE, concretely

What it is not

How it works

The demo, and its point

Configuration

CI

Security

Limitations

Docs

Install

Commands

Try it on real rules

Development

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

NULLFIRE

Detection Matchability Analyzer

Zero alerts means one of two things

60-second start

What it tells you

DEGRADED vs NULLFIRE, concretely

What it is not

How it works

The demo, and its point

Configuration

CI

Security

Limitations

Docs

Install

Commands

Try it on real rules

Development

About

Topics

Resources

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages