Skip to content
BranchesTags

Repository files navigation

TRACER (tracer_ids)

CI Release Build CodeQL Python License Package

Audit-first intrusion-detection benchmarking with time-robust evaluation and leakage diagnostics.

Quickstart · Contract · Repo Map · Docs · Contributing

TRACER is a benchmark contract plus a Python toolkit for intrusion-detection evaluation. It standardizes the prediction unit, protocol surface, and reporting invariants; exposes shortcut-sensitive baselines and time_robust controls; and keeps public claims tied to reproducible commands, manifests, and artifacts.

TRACER is evaluation infrastructure. It is not a SOC deployment guarantee, not a generic concept-drift benchmark by default, and not a claim that one pooled metric transfers to every dataset or environment.

At A Glance

Surface Value
Repository kmccleary3301/tracer_ids
Python package tracer_ids
CLI tracer
Contract version tracer-contract-1.0
Prediction unit transaction window
Core protocols baseline, time_robust
Shortcut controls time-only baseline, dataset-ID baseline
Primary audience researchers, ML engineers, security practitioners, OSS contributors
Support floor Python 3.10+
License MIT

Status

Signal State Notes
CI smoke 🟢 CLI help, claims validation, portability checks, dry-run pipeline wiring
Release build 🟢 Wheel + sdist build workflow
Code scanning 🔵 Manual CodeQL workflow is present
Repro extended 🔵 Container smoke plus optional self-hosted GPU lane
Public claims discipline 🟢 Claims map, benchmark contract, manifest validation, release checklist

Contents

Why TRACER Exists

Most IDS benchmark tables compress a large amount of protocol and data-hygiene complexity into one number. TRACER treats that as a methodological problem, not just a presentation problem.

TRACER is built around three commitments:

  1. Protocol is part of the result. Baseline and time_robust runs are different evaluation conditions, not cosmetic variants.
  2. Shortcut capacity should be measured directly. Time-only and dataset-ID baselines are explicit reporting surfaces, not private diagnostics.
  3. Claims should be auditable. Public-facing results should map back to commands, configs, manifests, and artifact paths.

Flow

flowchart LR
    A[Raw IDS sources<br/>PCAP / flows / CSV] --> B[Dataset parsers<br/>windowized records]
    B --> C[TRACER prediction unit<br/>transaction window]
    C --> D[Protocols<br/>baseline / time_robust]
    D --> E[Metrics and reports<br/>pooled + per-dataset]
    E --> F[Claims surface<br/>contract + manifests + artifacts]
Loading

What The Repository Includes

Area Included here Why it matters
Benchmark contract docs/benchmark_contract.md Fixes semantics for protocols, metrics, prediction unit, and non-claims
CLI workflows tracer_ids/cli.py Gives a small, stable surface for validation and dry-run usage
Data pipeline tracer_ids/data/ and tracer_ids/dataset_parsers/ Parses heterogeneous datasets into TRACER-compatible window records
Model implementations tracer_ids/models/ Baseline neural model family support and training-time integration
Tabular tooling tracer_ids/tabular/ Non-neural baselines and tabular feature paths
Training orchestration tracer_ids/training/ Pretraining, fine-tuning, metrics logging, manifest-aware runs
Validation scripts scripts/ci/ and scripts/validate_claims_map.py Prevents silent drift in paths, manifests, and claims structure
Paper/figure utilities scripts/figures/ and scripts/final_sprint/ Rebuilds aggregate reports, figures, and artifact ledgers

First Five Minutes

If you only want to verify the repo surface without datasets or expensive runs:

/usr/bin/miniconda3/bin/pip install -e .[dev]

tracer quickstart
tracer validate-claims
make portability-check
make manifest-check

Expected quickstart output is intentionally lightweight:

[ConfigEnv] resolved ... token(s) ...
[PipelineProfile] python pretrain.py --config ... --precompute-transforms all=false

That output means the wiring path is intact and the repo can resolve config tokens, route through the profile wrapper, and stop short of expensive execution.

What this quickstart does not do
  • It does not download datasets.
  • It does not run model training.
  • It does not certify benchmark correctness beyond the validated dry-run surface.
  • It does verify that the public CLI, config resolution, and portability/claims guardrails are wired coherently.

Installation

The most reliable local path today is a conda-backed Python environment with an editable install.

/usr/bin/miniconda3/bin/pip install -e .[dev]
python -m tracer_ids.cli --help
tracer --help

Environment variables

Variable Status Purpose
TRACER_DATA_ROOT preferred Dataset root override for portable orchestration scripts
TRACER_CACHE_ROOT preferred Cache root override
TRACER_CONDA_SH preferred Conda activation script path
TRACER_CONDA_ENV preferred Conda environment name
MNI_DATA_ROOT legacy alias Backward-compatible dataset root override
MNI_CACHE_ROOT legacy alias Backward-compatible cache override
MNI_CONDA_SH legacy alias Backward-compatible conda script path
MNI_CONDA_ENV legacy alias Backward-compatible conda environment name

Example:

export TRACER_DATA_ROOT=/data/network_anomaly_datasets
export TRACER_CACHE_ROOT="$TRACER_DATA_ROOT/.tracer_cache"
export TRACER_CONDA_SH=/usr/bin/miniconda3/etc/profile.d/conda.sh
export TRACER_CONDA_ENV=NP_6

For more detail, see docs/install.md.

Common Workflows

Goal Command Output
Dry-run the portable pipeline path tracer quickstart Prints the orchestrated command path without running training
Validate public claim block structure tracer validate-claims Checks the claims schema in docs/final_sprint/CLAIMS.md
Guard portable entrypoints against local path leakage make portability-check Fails on hardcoded-path regressions in portable shells/scripts
Validate discovered run manifests make manifest-check Checks manifest YAML structure when manifests are present
Run the profile wrapper directly python scripts/run_pipeline_profile.py --profile dev --dry-run Exercises config resolution and launch wiring
Build aggregate result JSONs make aggregates Produces aggregate benchmark summaries under logs/final_runs/
Build figure panels make figures-core Recreates core paper figures into the configured targets directory
Build artifact ledger make artifact-ledger Regenerates artifacts_final/ledger/ outputs

Persona-specific entry points

If you are... Start here
Researcher checking scope and claims docs/getting_started.md, docs/benchmark_contract.md, docs/final_sprint/CLAIMS.md
ML engineer validating local wiring docs/quickstart.md, docs/micro_run.md, docs/reproducibility.md
Security practitioner auditing benchmark claims docs/faq.md, docs/claims/README.md, docs/benchmark_contract.md
Contributor adding datasets or improving tooling CONTRIBUTING.md, docs/cookbook_add_dataset_parser.md, docs/engineering_guide_data_pipeline.md

CLI Reference

TRACER keeps the top-level CLI intentionally small:

Command What it does
tracer ci-verify Runs the local CI smoke wrapper
tracer profile Passes through to scripts/run_pipeline_profile.py
tracer quickstart Runs the portable dry-run quickstart
tracer validate-claims Validates claims markdown structure
tracer portability-check Runs hardcoded-path portability checks
tracer manifest-check Validates run-manifest YAML structure

Example:

tracer profile --profile dev --dry-run

Benchmark Contract And Reporting Model

The contract is not an appendix detail. It is part of the repository surface.

Contract surface TRACER policy
Prediction unit transaction window
Canonical label type
Native label retention true_label when available
Mandatory pooled reporting yes
Mandatory per-dataset reporting yes
Negative controls required where claim-relevant
Public non-claims explicit and versioned

Protocols

Protocol Intent Boundary
baseline Standard split/evaluation path Reference condition, not the only condition
time_robust Stress-tests schedule-sensitive behavior Not a generic concept-drift guarantee

Metrics

Task Primary metric Reporting requirement
Classification macro-F1 pooled + per-dataset summaries
Anomaly detection AUPRC prevalence caveats must remain explicit

Canonical references:

Repository Map

tracer_ids/
├── cli.py                  # small public CLI surface
├── data/                   # loaders, transforms, samplers, caching helpers
├── dataset_parsers/        # dataset-specific ingestion/parsing
├── ids/                    # IDS-specific types, parsing, scoring helpers
├── models/                 # neural baselines and model wrappers
├── tabular/                # tabular feature extraction / baselines
├── training/               # pretraining, fine-tuning, metrics logging
├── utils/                  # manifests, env resolution, profiling, cache utilities
└── misc/legacy_root/       # migration shims and legacy compatibility helpers

scripts/
├── ci/                     # repo guards for portability and manifests
├── figures/                # figure generation and composition
├── final_sprint/           # artifact ledger and thesis/paper helpers
├── helpers/                # shell bootstrap helpers
├── hpo/                    # HPO support code
├── ops/                    # portable shell entrypoints
├── overnight/              # long-run orchestration helpers
└── pcap/                   # PCAP preprocessing utilities

configs/                    # benchmark, dataset, baseline, and HPO configs
docker/                     # Snort/Suricata/Zeek support assets
docs/                       # benchmark contract, install, quickstart, claims, guides
tests/                      # unit and migration/repro checks
.github/workflows/          # CI, release build, repro-extended, CodeQL
What is intentionally not in the tracked repository surface
  • Large run logs and local caches under run_logs*/, logs/, and cache directories.
  • Scratch planning material and temporary writeups under docs_tmp/.
  • Machine-local thesis and artifact workspaces that are reproducible or too environment-specific to version directly.

Code And Module Guide

Path Focus
tracer_ids/data/ Data loading, caching, transforms, split-time behavior
tracer_ids/dataset_parsers/ Dataset-specific parse layers and conversion into TRACER-compatible records
tracer_ids/models/ Neural baseline implementations and wrappers
tracer_ids/tabular/ Tabular feature extraction and benchmark support
tracer_ids/training/ Training loops, metrics registry, W&B integration
tracer_ids/utils/run_manifest.py Lightweight manifest generation for reproducibility tracking
scripts/run_pipeline_profile.py Main profile wrapper for dry-runs and scoped execution
scripts/validate_claims_map.py Claims schema enforcement
scripts/ci/check_hardcoded_paths.py Portable path guard
scripts/ci/validate_run_manifests.py Manifest structure validator

Documentation Guide

The repository now has enough surface area that a docs index is useful. Start with docs/README.md for a categorized documentation map.

High-signal documents

Topic Start here
Install and local setup docs/install.md
Dry-run verification docs/quickstart.md
Scoped small-run path docs/micro_run.md
Benchmark semantics docs/benchmark_contract.md
Reproducibility policy docs/reproducibility.md
Claims mapping docs/claims/README.md
Contribution path docs/cookbook_add_dataset_parser.md
Voice and messaging constraints docs/brand_voice.md
Launch/release operations docs/launch_ops.md, docs/release_checklist.md

Reproducibility, Claims, And Release Discipline

TRACER treats release quality as part of technical quality.

Guardrail Enforced by
Claims structure tracer validate-claims
Portable shell entrypoints make portability-check
Manifest structure make manifest-check
Release packaging GitHub Actions release-build workflow
Container / optional GPU repro lanes GitHub Actions repro-extended workflow

Release-facing files

Examples

1. Portable no-data validation path

tracer quickstart
tracer validate-claims
make portability-check
make manifest-check

2. Profile dry-run with an explicit dataset root

TRACER_DATA_ROOT=/path/to/datasets \
python scripts/run_pipeline_profile.py --profile dev --dry-run

3. Scoped micro-run path

TRACER_DATA_ROOT=/path/to/datasets \
python scripts/run_pipeline_profile.py \
  --profile dev \
  --pretrain-arg=--sample-multiplier \
  --pretrain-arg=0.01

4. Rebuild aggregate JSON summaries

make aggregates

5. Rebuild claims-adjacent figures

make figures-core
make compose
make artifact-ledger
More command surfaces worth knowing about
Command / target When to use it
tracer ci-verify local CI-style smoke wrapper
tracer profile --profile dev --dry-run direct access to the profile orchestrator
make aggregates regenerate aggregate JSON summaries
make figures-core regenerate core figure panels
make compose compose multi-panel figures
make artifact-ledger rebuild small tracked artifact ledgers

Contribution And Project Policy

Before opening a PR, run the smallest relevant validation surface you can justify.

Change type Expected evidence
Docs / messaging only link check + relevant markdown sanity
CLI / portability changes tracer quickstart, make portability-check
Claims or release surface changes tracer validate-claims, make manifest-check
Loader / pipeline orchestration changes dry-run profile path plus targeted script checks

Primary references:

Citation, License, And Security

Topic Location
Citation metadata CITATION.cff
License LICENSE
Security policy SECURITY.md
Code of conduct CODE_OF_CONDUCT.md

If you are looking for the shortest navigation path:

About

TRACER: audit-first intrusion-detection benchmarking toolkit with time-robust evaluation and leakage diagnostics.

Topics

benchmarkcybersecuritydata-hygieneintrusion-detectionmachine-learningnetwork-securityreproducibilitytime-robust-evaluation

Resources

Readme
MIT license

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy
Activity

Stars

0 stars

Watchers

1 watching

Forks

1 fork
Report repository

Releases

Packages

Contributors

Languages