Shinobi

Silent Web Scraper — Anti-Blocking Download System

Language / Idioma
English | Español

XWA submodule focused on silent web scraping with anti-blocking — under active development

---

Overview · Capabilities · Anti-Blocking · Quick Start · Launch Script · Docker Compose · Standalone · Modes · Env Variables · Docs

---

Overview

Shinobi is a stealth web scraper with a single web interface. It downloads entire sites — HTML, CSS, JS, images, PDFs — while evading detection through multiple anti-blocking layers.

Interface	Directory	Language	Type
Shinobi Web	/ (monorepo root)	Rust (Axum) + Angular 19	Web application (standalone or Docker)

Capabilities

• Recursive Crawling — BFS page discovery with configurable depth and max pages
• JavaScript Rendering — Headless Chromium engine for SPA/React/Vue/Angular sites
• Asset Download — HTML, CSS, JS, images, PDFs, archives, media, fonts
• File Type Filtering — Select which file extensions to download
• Same-Domain Scoping — Stay within target domain or crawl freely
• Real-Time Progress — SSE live stream with pages scraped, files downloaded, current URL
• Downloaded File Browser — Browse and open downloaded files from the web UI
• Two Operation Modes:
  • Fast Test — Pure Rust, zero external deps, crawl + download + anti-blocking
  • Deep Research — Python sidecar adds structured data extraction (JSON-LD, microdata, Open Graph, RDFa), NLP analysis (summary, entities, keywords), custom CSS selectors, and enriched email/phone extraction

Anti-Blocking System

• User-Agent Rotation — 15 real browser UAs (Chrome, Firefox, Safari, Edge, mobile)
• Header Randomization — Accept, Accept-Language, Sec-CH-UA, Sec-Fetch-* per request
• Request Delay + Jitter — Configurable base delay with random jitter
• Exponential Backoff — Retry with jitter on failures (configurable attempts)
• Rate Limit Handling — Detects HTTP 429/503, waits, retries with longer backoff
• Proxy Support — HTTP/HTTPS/SOCKS5 proxy rotation

---

Quick Start

Launch Script (Recommended)

./shinobi.sh --fast       # Rust backend + frontend (Fast Test mode)
./shinobi.sh --deep        # Rust + Python extractor (Fast Test + Deep Research)
./shinobi.sh --docker      # Everything via docker-compose
./shinobi.sh --help        # Full usage help

Web Version (Docker Compose)

docker compose up -d --build

• Web UI: http://localhost:8080
• Extractor API: http://localhost:9090 (Deep Research only)

Standalone (No Docker)

Fast Test mode (Rust only)
cargo run --release
Full stack (Rust + Python extractor)
./shinobi.sh --deep

Web UI at http://localhost:8080. Downloaded files stored in ./downloads/.

Execution Modes

Mode	Command	Description
Fast Test	./shinobi.sh --fast	Crawling + asset download + anti-blocking. No external dependencies.
Deep Research	./shinobi.sh --deep	Adds Python sidecar for structured data extraction (JSON-LD, microdata, OG), NLP analysis (summary, entities, keywords), and custom CSS selectors.
Docker	./shinobi.sh --docker	Both services via docker-compose.
Python Only	./shinobi.sh --python-only	Extractor standalone for development.

Environment Variables

Variable	Default	Description
PORT	8080	HTTP listen port (Rust)
DATA_DIR	downloads	Downloaded files directory
RUST_LOG	shinobi=info,tower_http=info	Logging verbosity
EXTRACTOR_URL	http://localhost:9090	Python extractor endpoint (Deep Research)

---

Related Documents

Document	Description
docs/manual.md	Development and production deployment guide
docs/ui-architecture.md	Frontend architecture specification
ROADMAP.md	Development phases and milestones
docs/project-structure.md	Detailed codebase structure with file-by-file breakdown and API reference

---

X

& &