🐦 logpeck — Forensic MongoDB Log Analytics

Surgical performance discovery for MongoDB. Reconstruct the "Why" behind latency, failures, and architectural bottlenecks.

---

🎯 What you get from LogPeck

Standard monitoring tells you when things are slow. LogPeck tells you why. It transforms raw MongoDB logs into actionable forensic insights:

• 🐢 Latency Cliff Discovery: Identifies query shapes where the slowest samples are significantly slower than the average—revealing intermittent resource blocking.
• 🚨 Architectural Anti-Patterns: Automatically flags COLLSCAN events, in-memory sorts, and inefficient index usage.
• 🔗 Join & Regex Visibility: Surface non-standard $lookup stages and CPU-intensive regex scans that impact cluster stability.
• 🛡️ Infrastructure Truth: Isolates systemic network errors (timeouts, disconnects) from business workload performance.
• 🔌 Application Attribution: Correlates every slow query to a specific Application, IP, and User, even in logs where that context is missing.
• 📉 Volumetric AAS Load: Visualizes the physical "weight" of every operation relative to the total cluster load.

---

📦 Installation

pip3 install git+https://github.com/tanujbolisetty/mongodb-logpeck.git

Tip

If the peck command is not found after installation, ensure your Python binary directory is in your PATH. For example, on macOS: export PATH=$PATH:$(python3 -m site --user-base)/bin

---

⚡ Quick Start (How to use)

Generate a professional, six-tab forensic report in seconds:

# Analyze a log and generate an interactive dashboard
peck dashboard --file mongod.log --html forensic_report.html

Or perform surgical analysis directly in your terminal:

# Analyze business workload hotspots
peck workload --file mongod.log

# Analyze all systemic errors and timeouts
peck failure-workload --file mongod.log

---

🛠️ Surgical CLI Usage

The peck command provides multiple forensic lenses into your cluster data.

1. Interactive 6-Tab Forensic Dashboard

The flagship feature of logpeck. Generates a professional six-tab surgical report for exhaustive diagnostic review.

# Standard Production Analysis (GZip Support)
# Defaults to --latency 0 (Full forensic capture)
peck dashboard --file mongod.log.gz --html output/dashboard.html

# Bulk Folder Analysis
# Generates a separate report for each log file (e.g. `mongod.log_report.html`) in the output directory
peck dashboard --folder /path/to/log/directory --html output/

# Custom Diagnostic Sizing (Optional: filter for queries > 100ms)
peck dashboard --file mongod.log.gz --latency 100

2. Workload Forensics (Business Hotspots)

Deep-dive into business-level performance hotspots while excluding system noise.

# Analyze business workload (Defaults to --latency 0)
peck workload --file mongod.log.gz

# Analyze business workload with a 500ms filter
peck workload --file mongod.log.gz --latency 500

3. System Query Forensics (Infrastructure)

Analyze background diagnostics and infrastructure telemetry (TTL cleanup, Oplog truncation, Index builds).

# Analyze infrastructure task performance
peck system-workload --file mongod.log.gz

4. Failure & Timeout Forensics

Analyze systemic failures, timeouts, and error hotspots.

# Analyze all failures and timeouts
peck failure-workload --file mongod.log.gz

5. Connection Portfolio & Client Analysis

Identify connection churn, authentication failures, and app attribution.

peck connections --file mongod.log.gz

6. Surgical Filtering

Structured multi-dimensional forensics using logical AND chaining.

# Filter by latency (> 500ms)
peck filter --file mongod.log --filters '{"ms": {"gt": 500}}'

# Filter by namespace and operation
peck filter --file mongod.log --filters '{"ns": "production.orders", "op": "update"}'

# Filter by plan (COLLSCAN)
peck filter --file mongod.log --filters '{"plan": "COLLSCAN"}' --cards

# Rapid volume check (Count only)
peck filter --file mongod.log --filters '{"ms": {"gt": 1000}}' --count

# Control display volume (Top 5 results)
peck filter --file mongod.log --filters '{"ms": {"gt": 500}}' --limit 5

# Deep-path filtering (For non-standard or nested fields)
peck filter --file mongod.log --filters '{"attr.storage.data.txnBytesDirty": {"gt": 536045710}}' --cards

Supported Operators

Operator	Description	Example
gt	Greater than (numeric)	'{"ms": {"gt": 500}}'
lt	Less than (numeric)	'{"ms": {"lt": 100}}'
contains	Substring match (case-insensitive)	'{"ns": {"contains": "orders"}}'
eq	Equal to (default, case-insensitive string)	'{"plan": "COLLSCAN"}' or '{"plan": {"eq": "COLLSCAN"}}'

Important

Windows Shell Note (PowerShell & CMD): Windows command line parsers can strip or misinterpret nested single/double quotes, leading to a JSONDecodeError. If you are on Windows, format your filters as follows:

• PowerShell: Escape the inner quotes: --filters '{\"ms\": {\"gt\": 500}}'
• CMD: Wrap in double quotes and escape the inner quotes: --filters "{\"ms\": {\"gt\": 500}}"

Common Filter Keys (Shortcuts)

LogPeck standardizes and flattens over 40+ MongoDB fields. The most common shortcuts you can query against are:

Key	Description	MongoDB Source Fields
ms	Operation duration in milliseconds	durationMillis, durationMS
op	Operation type (e.g., find, update, insert, getmore)	attr.command keys / type
ns	Namespace of the target collection	attr.ns
app_name	Client application name	attr.appName, doc.application.name
user	Database user identity	attr.user
client_ip	Client IP address	attr.remote / client
query_hash	Query fingerprint hash	attr.queryHash
query_shape_hash	Query shape fingerprint hash	attr.queryShapeHash
plan_cache_key	Plan cache key	attr.planCacheKey
has_regex	Boolean: query contains a $regex parameter	Derived
has_lookup	Boolean: query contains a $lookup aggregation	Derived
keysExamined	Number of index entries scanned	attr.keysExamined
docsExamined	Number of documents scanned from disk/memory	attr.docsExamined
nreturned	Number of documents returned	attr.nreturned
reslen	Size of the result set in bytes	attr.reslen
nModified	Number of documents modified	attr.nModified
ndeleted	Number of documents deleted	attr.ndeleted
storage_wait	Derived physical storage wait (ms)	read + write + cache stalls
lock_wait	Lock contention acquisition wait time (us)	attr.locks.*.timeAcquiringMicros
queue_wait	Time spent waiting for execution ticket (us)	queues.execution.totalTimeQueuedMicros
replication_wait	Replication flow control throttle wait (ms)	attr.flowControlMillis

Tip

Deep-Path Filtering: You can also filter on any non-standard or raw MongoDB JSON field by specifying its exact path, e.g. '{"attr.storage.data.txnBytesDirty": {"gt": 50000}}'. Check the Reference Tab in the dashboard to see all 40+ mapping definitions.

7. Forensic Search (Stateful vs. Stateless)

LogPeck offers two ways to discover information:

• Forensic Search (Default): Optimized stateful search. Instead of searching the raw log text, it queries only constructed/extracted metadata fields (specifically the message, connection context, app identity, namespace, query command parameters, and query hashes). This enables it to backfill identities, finding slow queries associated with a client app even if the app name is not written on that specific log line.
• High-Precision Search (--grep): A stateless, full-text match. Mimics standard grep speed and behavior by searching the entire raw JSON string of the log entry.

# Forensic (Metadata & Identity-Backfilled): Find everything connected to the identity
peck search --file mongod.log --keyword "compass"

# High-Resolution: Show raw log lines and full timestamps
peck search --file mongod.log --keyword "compass" --full

# High-Precision (Grep): Search the entire raw JSON log text for literal matches
peck search --file mongod.log --keyword "compass" --grep

---

🗺️ CLI vs Dashboard Mapping

Use the table below to find the surgical CLI command equivalent for each professional dashboard tab.

Dashboard Tab	CLI Command	Purpose	Key Options
1. Global Health	peck health	High-level summary of severity levels and components.	--json
2. Business Workload	peck workload	Analyzes application-level slow queries.	--latency, --json
3. System Workload	peck system-workload	Analyzes infrastructure tasks (TTL, Oplog).	--latency, --json
4. Failure Forensics	peck failure-workload	Analyzes systemic timeouts and error codes.	--latency, --json
5. Connection Analytics	peck connections	Profiles client apps and connection churn.	--json
6. Reference	(Automatic)	Registry of metrics and rules.	N/A
-	peck search	Surgical keyword forensic search.	--keyword, --grep, --full, --limit, --count
-	peck filter	Multi-dimensional forensic filtering.	--filters, --full, --limit, --count
-	peck dashboard	Generates the full 6-tab HTML dashboard.	--file, --folder, --latency, --html

---

🧬 Forensic Matrix (6-Tab Dashboard)

The HTML report is organized into six professional focus areas:

Tab	Focus	Key Diagnostics
🏥 Health Overview	Global Fleet Pulse	Cluster-wide severities, components, and primary bottleneck.
🛠️ System Query Forensics	Infrastructure Ops	TTL Cleanup, Oplog, Background Index Builds, and Admin tasks.
🐢 Business Workload	Performance Hotspots	Detailed analysis of user-level query shapes and latency cliffs.
🚨 Failure Forensics	Workload Interruptions	Consolidated Workload Errors (Lethal) and Timeouts (MaxTimeMS).
🔌 Connection Analytics	Application Hygiene	Identity attribution, connection churn, and driver fingerprinting.
📚 Reference	Diagnostic Glossary	Dynamic rule definitions, thresholds, and technical descriptions.

---

👤 Author

Tanuj Kumar Bolisetty
GitHub: @tanujbolisetty

---

📄 License

MIT © 2026 Distributed under the MIT License. See LICENSE for more information.