🕷️ Spider

An ultra-fast image scraper built in Rust, leveraging asynchronous programming and multi-threading for unmatched performance.
Spider crawls a website and downloads all its images, with an optional recursive mode to explore multiple depth levels.

---

🚀 Features

• High-performance systems language: Rust
• Asynchronous powered by Tokio to handle thousands of network requests in parallel
• Safe multi-threading with Arc, Mutex, DashSet, and semaphores to prevent race conditions
• Parallel downloads optimized with reqwest and futures
• Duplicate handling to avoid unnecessary downloads
• Configurable recursive scraping (depth limit)
• Robust: handles network errors and various image formats

---

📦 Installation

Prerequisites

• Rust installed (2021 edition or later)
• cargo available in your terminal

Clone and build

git clone https://github.com/Frqnku/spider.git
cd spider
make build

The spider binary will be generated at the project root.

---

🛠 Usage

Basic command

./spider <URL>

Available options

Option	Description	Default
-r, --recursive	Enable recursive scraping	false
-l, --limit <N>	Max scraping depth (requires --recursive)	5
-p, --path <PATH>	Directory to save images	./data
<URL>	The URL of the page to scrape	required

Examples

# Scrape all images from a page
./spider https://example.com

# Recursively scrape up to 3 levels deep
./spider -r -l 3 https://example.com

# Specify a destination folder
./spider -p ./images https://example.com

---

⚙️ How it works

1. CLI argument parsing with clap
2. Disk access checks to ensure a valid and writable directory
3. Asynchronous downloads with reqwest and futures
4. Image & link extraction with a simple HTML parser
5. Concurrency management using:
   • Arc + Mutex to share state (visited)
   • DashSet for fast duplicate tracking
   • Semaphore to limit the number of concurrent tasks
6. Safe file saving: each image is saved with a unique name based on UUID and timestamp

---

📚 Tech Stack

• Language: Rust (2021 edition)
• Networking: reqwest
• Async runtime: Tokio
• Concurrency: Arc, Mutex, Semaphore, DashSet
• URL parsing: url
• Unique IDs: uuid
• Timestamps: chrono
• CLI: clap

---

📜 License

This project is licensed under the MIT License.
See the LICENSE file for details.

---

💡 Spider is designed to maximize performance while remaining simple to use. Thanks to Rust and its safe async/multi-threaded architecture, it can handle a huge number of simultaneous downloads without overwhelming your system.