os-intrusion-detection-content-at-antiphishing: support external Suricata dataset files

[julioliraup]

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
• I have searched the existing issues, open and closed, and I'm convinced that mine is new.
• The title contains the plugin to which this issue belongs

Describe the bug
The Antiphishing IDS/IPS content plugin currently installs the main
antiphishing.rules file, but the ruleset also depends on two external
Suricata dataset files:

• phishing.lst
• phishing_ips.lst

The rules reference these files using Suricata's dataset load syntax:

dataset:isset,phishing_domains,type string,load phishing.lst

and:

dataset:isset,phishing_ips,type ipv4,load phishing_ips.lst

On OPNsense 26.7.2_2, the .rules file is installed under:

/usr/local/etc/suricata/opnsense.rules/

but the dataset files are not made available there.

As a result, Suricata reports:

failed to set up dataset 'phishing_domains'

and:

failed to set up dataset 'phishing_ips'

The static signatures continue to load, but the dataset-backed signatures are inactive.

This appears to be an integration issue between the OPNsense IDS content updater and a Suricata ruleset that references external dataset files.

We are currently investigating the correct OPNsense-supported mechanism for distributing these auxiliary files and would prefer to follow the existing OPNsense architecture rather than introducing an Antiphishing-specific workaround.

To Reproduce

1. Install/enable os-intrusion-detection-content-at-antiphishing.
2. Enable the Antiphishing ruleset in Intrusion Detection.
3. Update/download the rules.
4. Reload Suricata.
5. Check the Suricata logs.
6. Observe errors for the phishing_domains and phishing_ips datasets.

The installed ruleset contains references to:

load phishing.lst

and:

load phishing_ips.lst

but the corresponding files are not available in the directory from which the ruleset is loaded.

Expected behavior
The plugin should install all files required by the Antiphishing ruleset, not only antiphishing.rules.

The expected installation is conceptually:

antiphishing.rules
phishing.lst
phishing_ips.lst

with the dataset files available from the path expected by Suricata's
relative load directive.

Screenshots

Relevant log files

Error -- failed to set up dataset 'phishing_domains'.
Error -- error parsing signature "alert dns $HOME_NET any -> any any
(msg:"AT DNS query to suspicious domain - Phishing"; dns.query;
dataset:isset,phishing_domains,type string,load phishing.lst; ...
sid=6000000...)" from file
/usr/local/etc/suricata/opnsense.rules/antiphishing.rules at line 6

Error -- failed to set up dataset 'phishing_ips'.
Error -- error parsing signature "alert ip $HOME_NET any -> any any
(msg:"AT IP dataset - Phishing"; ip.dst; dataset:isset,phishing_ips,
type ipv4,load phishing_ips.lst; ... sid=6000002...)" from file
/usr/local/etc/suricata/opnsense.rules/antiphishing.rules at line 10

on:

/usr/local/etc/suricata/opnsense.rules/antiphishing.rules

Additional context
The Antiphishing ruleset is designed to be consumed by Suricata as a
portable ruleset and is also integrated with other Suricata ecosystems.

The dataset files are intentionally kept separate from the .rules file
because they are Suricata datasets rather than Suricata signature files.

We are investigating whether the OPNsense plugin metadata already
provides a supported mechanism for declaring multiple files belonging to
the same ruleset/source, or whether support for auxiliary dataset files
needs to be added to the OPNsense IDS content infrastructure.

We would prefer to preserve the standard Suricata dataset layout rather
than rename dataset files to .rules solely to satisfy an OPNsense
file-extension convention.

Suricata-update itself has explicit handling for files that do not end in
.rules and for dataset filenames:

https://github.com/OISF/suricata-update/releases

In particular, the 1.3.1 changelog mentions:

• handling URLs of bare files that don't end in .rules;
• detecting dataset load;
• preserving dataset filenames;
• assigning unique filenames to source files, specifically including dataset files.

We are therefore trying to determine the appropriate OPNsense integration
pattern before making changes to the Antiphishing distribution format.

Environment

OPNsense 26.7.2_2
Architecture: amd64
Plugin: os-intrusion-detection-content-at-antiphishing
Suricata IDS
Interface: LAN
Mode: IDS (not IPS)

[julioliraup]

Update: testing the .rules extension workaround

I also tested an alternative approach of renaming the dataset files from:

• phishing.lst → phishing.rules
• phishing_ips.lst → phishing_ips.rules

and updating the corresponding load directives.

This approach is not suitable.

Although the files can be made visible to the OPNsense ruleset installation pipeline by using the .rules extension, OPNsense then treats the dataset files as Suricata rule files during ruleset processing.

Since these files contain dataset values rather than Suricata signatures, this caused the IDS ruleset processing/reload to fail during testing.

This confirms that simply changing the dataset filename extension is not an appropriate workaround.

The desired behavior is therefore to keep the files as dataset files:

antiphishing.rules
phishing.lst
phishing_ips.lst

and have the OPNsense content integration install the auxiliary dataset files without processing them as signature files.

I'm continuing to investigate the existing OPNsense metadata/update mechanisms to determine the appropriate way to represent these auxiliary files.

[AdSchellevis]

There isn't a uniform way to ship datasets I'm afraid, it always requires configuration, which is also the reason that vendors like Proofpoint currently don't use them. If the set depends on them and the rules can't be rewritten, we likely better remove the plugin from our repository so you can experiment a bit further on your end without risking shipping broken rulesets to users.

[julioliraup]

Keep it as it is for now; the HTTP signatures are working correctly, but the DNS, TLS and IPv4 dataset-based signatures are not being loaded by the current OPNsense integration.

I have tested the .rules workaround as well, but it is not suitable because OPNsense treats the dataset files as rule files.

I'll investigate another approach that does not require changing the semantics of the Suricata datasets.

Also, is there any intention to add native support for external Suricata datasets in the IDS content framework? If this is something the OPNsense team would consider, I'd be happy to prepare a PR for it.

[julioliraup]

Update: I have now prepared an alternative OPNsense-specific ruleset and opened a PR for testing.

The new antiphishing-opnsense.rules keeps the existing HTTP signatures and adds the IPv4 intelligence without relying on external .lst dataset files. DNS/TLS dataset-based signatures are intentionally excluded for now because converting the 241k-domain dataset into conventional signatures caused Suricata to exhaust available memory during startup in my OPNsense test environment.

I am keeping the upstream Antiphishing ruleset unchanged, including its native DNS/TLS datasets.

I'll continue investigating a proper solution for dataset support separately. For now, the new PR is intended to provide a stable and testable OPNsense integration rather than shipping a known-broken dataset configuration.

One question for the OPNsense team: would it be acceptable for the plugin to use a small auxiliary script, integrated with the existing update mechanism, to download/update and manage the required Suricata dataset files separately from the .rules files?

If this is an acceptable approach within the OPNsense architecture, I can investigate and prepare an implementation for it. I would prefer to use an existing OPNsense-supported mechanism where possible rather than introducing project-specific behavior.

[julioliraup]

Update: I have prepared and opened a PR with a tested interim solution for the current dataset integration problem.

The approach keeps the native Suricata datasets (phishing.lst and phishing_ips.lst) instead of renaming them to .rules or converting their contents into signatures.

I found that the OPNsense content updater can download the auxiliary files when they are declared in the ruleset metadata. The current implementation then loads them from the OPNsense rule staging directory using an OPNsense-specific ruleset variant.

This is an interim solution based on the current OPNsense behavior, not a proposal for a generic dataset mechanism. I would still be interested in a proper, framework-level solution for external Suricata datasets in the future.