Vectorscope

A study/research tool for cybersecurity that lets you query MITRE ATT&CK techniques, CVE descriptions, and threat actor profiles in natural language. Built with a RAG (Retrieval-Augmented Generation) pipeline, it retrieves relevant threat intelligence documents and synthesizes clear, sourced answers.

Instead of ctrl+F-ing through the ATT&CK framework or scrolling through NVD listings, ask questions like:

• "What techniques does APT29 use for lateral movement?"
• "Explain T1059 in simple terms"
• "What CVEs are related to credential dumping?"
• "Compare phishing techniques across different threat groups"

How It Works

1. Ingest — Downloads MITRE ATT&CK data (techniques, groups, software, mitigations) and CVE descriptions from the NVD API, saves them as JSON
2. Embed — Converts documents into vector embeddings using sentence-transformers and indexes them in ChromaDB
3. Query — Finds the most relevant documents for your question using hybrid search (exact ID match + semantic similarity)
4. Synthesize — An LLM reads the retrieved context and generates a sourced, plain-language answer

Setup

Requirements: Python 3.10+

# Clone and install
git clone https://github.com/faisalhossainnyc/vectorscope.git
cd vectorscope
pip install -r requirements.txt

# Download data and build the vector index (takes a few minutes)
python main.py ingest

For LLM synthesis, you need at least one backend available (see below). Ollama is recommended — it's free and runs locally.

# Install Ollama from https://ollama.ai, then:
ollama pull llama3.2

Usage

# Search for relevant documents (no LLM needed)
python main.py search "lateral movement techniques"

# Filter by document type
python main.py search "credential access" --type technique

# Ask a question with LLM-synthesized answer
python main.py ask "What techniques does APT29 use for credential access?"

# Ask with type filter
python main.py ask "What CVEs affect Apache?" --type cve

# Fetch a single document by exact ID
python main.py fetch T1059
python main.py fetch G0016
python main.py fetch CVE-2021-44228

# Check what's in the index
python main.py stats

Type filter values: technique, group, software, cve, mitigation, kev

LLM Backends

The app detects which backend to use automatically, checked in this order:

Priority	Backend	Setup	Cost
1	Ollama (default)	Install from ollama.ai → ollama pull llama3.2	Free
2	Google Gemini	export GEMINI_API_KEY='...'	Free tier available
3	Anthropic Claude	export ANTHROPIC_API_KEY='...'	Paid
4	OpenAI	export OPENAI_API_KEY='...'	Paid

If Ollama isn't running and no API keys are set, search still works — only ask requires a backend.

Data Sources

• MITRE ATT&CK — techniques, threat actor groups, software/tools, and mitigations (via STIX 2.1 data)
• NVD — CVE vulnerability descriptions with CVSS scores, severity ratings, and CWE mappings
• CISA KEV — Known Exploited Vulnerabilities catalog (~1,500 entries, high-signal subset of CVEs actively exploited in the wild)

Evaluating Retrieval Quality

The project includes a retrieval evaluation harness, a fixed set of 29 queries with known expected document IDs, scored with recall@k.

# Run the full eval (recall@1, @3, @5 with per-query breakdown)
make eval

# Or directly with custom k values
.venv/bin/python scripts/eval.py --k 1 3 5 10

Results are saved to docs/eval_results_<timestamp>.json. See docs/eval_baseline.md for baseline scores and known weak areas.

Current scores (Phase 2 final): recall@1 69.0%, recall@3 79.3%, recall@5 82.8%.

Project Structure

├── main.py                      # CLI entry point
├── requirements.txt
├── Makefile                     # setup / ingest / test / eval / lint targets
├── scripts/
│   ├── ingest_attack.py         # Downloads + parses MITRE ATT&CK STIX data
│   ├── ingest_cve.py            # Fetches CVEs from NVD API
│   ├── ingest_kev.py            # Fetches CISA Known Exploited Vulnerabilities catalog
│   ├── eval.py                  # Retrieval eval harness (recall@k scoring)
│   └── eval_dataset.py          # Fixed 29-query evaluation dataset
├── src/
│   ├── embedder.py              # ChromaDB indexing pipeline (cached per process)
│   ├── query_engine.py          # Hybrid retrieval + LLM synthesis
│   └── repl.py                  # Interactive REPL mode
├── docs/
│   ├── eval_baseline.md         # Phase 1 baseline scores + weak area analysis
│   └── eval_results_*.json      # Timestamped eval run outputs
└── data/                        # Downloaded data + vector index (gitignored)

Tech Stack

Python, ChromaDB, sentence-transformers (BAAI/bge-small-en-v1.5), Ollama / Gemini / Claude / OpenAI

Troubleshooting

NVD API 404 error during ingest NVD rejects date ranges over ~120 days in a single request. The incremental CVE updater is planned for a future phase. For now, your existing CVE data in the index is still queryable.

BertModel LOAD REPORT — UNEXPECTED key Harmless warning from a minor version mismatch in sentence-transformers. Has no effect on embedding quality.

Ollama not available search and fetch work without any LLM. Only ask needs a backend. Run ollama pull llama3.2 or export one of the API key env vars listed in the LLM Backends section.