MISP Blocklist Cache

A small containerized service for building cached text blocklists from MISP attribute searches and serving them over HTTP/S with OpenResty or nginx.

The goal is to avoid repeated MISP database/API hits from many downstream users. Instead, a scheduled builder queries MISP, writes newline-delimited .txt files into a cache directory, and the web server serves those cached files.

What it does

• Reads blocklist definitions from a YAML config file and dynamically builds required configurations.
• Runs MISP attribute searches using PyMISP on a cron schedule.
• Proxy authenticates user requests with their MISP API key.
• Serves cached files with OpenResty.

Runtime layout

Default paths used by the container:

/config/blocklists-config.yaml             Custom YAML location to supercede default
/config/blocklist-builder.cron             Custom cron file location to supercede default

/var/run/openresty/blocklists.yaml         Runtime blocklist config
/var/run/openresty/nginx.conf.d/           Dynamically generated nginx/OpenResty configs
/var/cache/openresty/blocklists/           Generated blocklist text files

Configuration

Blocklists are defined in YAML. Each item maps a public endpoint to one or more MISP search rules.

Example:

blocklists:
  - endpoint: "/ip/all"
    rules:
      - search_params:
          type:
            - ip-src
            - ip-dst
          last: "30d"
          to_ids: true
          enforceWarninglist: true

  - endpoint: "/ip/high"
    rules:
      - search_params:
          type:
            - ip-src
            - ip-dst
          last: "30d"
          to_ids: true
          enforceWarninglist: true
          tags:
            OR:
              - 'admiralty-scale:source-reliability="a"'
              - 'admiralty-scale:source-reliability="b"'
            AND:
              - 'admiralty-scale:information-credibility="1"'
      - search_params:
          type:
            - ip-src
            - ip-dst
          last: "30d"
          to_ids: true
          enforceWarninglist: true
          tags:
            OR:
              - 'admiralty-scale:source-reliability="a"'
              - 'admiralty-scale:source-reliability="b"'
            AND:
              - 'admiralty-scale:information-credibility="2"'

Each rule under a blocklist is queried separately. The final output file contains the union of all returned MISP attribute values for that blocklist.

The builder fetches each rule in pages of up to 100,000 attributes and continues until MISP returns an empty page. Query logs include the endpoint, rule, page, and the reason for any retry or rejection.

Included feeds

The default config defines these cached feeds:

Endpoint	Output file	MISP attribute types	Lookback	Notes
/ip/all	ip-all.txt	ip-src, ip-dst	30d	All IDS IP attributes after warninglist enforcement
/ip/high	ip-high.txt	ip-src, ip-dst	30d	High-confidence IP attributes based on Admiralty Scale tags
/domain/all	domain-all.txt	domain, hostname	60d	All IDS domain/hostname attributes after warninglist enforcement
/domain/high	domain-high.txt	domain, hostname	60d	High-confidence domain/hostname attributes based on Admiralty Scale tags
/url/all	url-all.txt	url	90d	All IDS URL attributes after warninglist enforcement
/url/high	url-high.txt	url	90d	High-confidence URL attributes based on Admiralty Scale tags
/md5/all	md5-all.txt	md5	365d	All IDS MD5 attributes after warninglist enforcement
/sha256/all	sha256-all.txt	sha256	365d	All IDS SHA256 attributes after warninglist enforcement

Endpoint to filename mapping

Endpoints are converted to filenames by stripping slashes, joining path segments with -, and appending .txt.

Examples:

/ip/all       -> ip-all.txt
/ip/high      -> ip-high.txt
/domain/all   -> domain-all.txt
/sha256/all   -> sha256-all.txt

Generated files are written to:

/var/cache/openresty/blocklists

Environment variables

Required:

MISP_API_KEY

MISP_FQDN

Optional:

MISP_VERIFY_TLS=true

Set MISP_VERIFY_TLS=false only for development or trusted internal environments where certificate verification cannot be used.

Scheduling

The container uses cron to run the blocklist builder on a schedule. Cron decides when the builder runs; the YAML file only defines what to build.

Example /etc/cron.d/blocklist-builder:

0 */3 * * * /usr/bin/python3 -u /usr/local/bin/misp_blocklist_builder.py

For larger deployments, specific feeds can also be staggered:

0 * * * * root /usr/local/bin/run-blocklist-builder.sh --endpoint /ip/all >> /proc/1/fd/1 2>> /proc/1/fd/2
10 * * * * root /usr/local/bin/run-blocklist-builder.sh --endpoint /domain/all >> /proc/1/fd/1 2>> /proc/1/fd/2
20 * * * * root /usr/local/bin/run-blocklist-builder.sh --endpoint /url/all >> /proc/1/fd/1 2>> /proc/1/fd/2
30 * * * * root /usr/local/bin/run-blocklist-builder.sh --endpoint /md5/all >> /proc/1/fd/1 2>> /proc/1/fd/2
40 * * * * root /usr/local/bin/run-blocklist-builder.sh --endpoint /sha256/all >> /proc/1/fd/1 2>> /proc/1/fd/2