deduper service

deduper is a Python-based tool and library for identifying and removing duplicate files across file systems or datasets. It leverages hashing algorithms and multithreading to efficiently scan directories, compare files, and manage duplicates. The project provides both a programmatic API and a command-line interface to facilitate flexible integration and easy usage for various deduplication scenarios.

---

Introduction

deduper is designed to help users efficiently detect and remove duplicate files. It recursively scans directories, computes file hashes for content comparison, and provides options for reporting or deleting duplicate files. The project is modular, allowing users to customize deduplication strategies and integrate its capabilities into other Python projects.

---

Requirements

Before installing or running deduper, ensure your environment meets these requirements:

• Python 3.7 or above
• Standard Python libraries: os, hashlib, concurrent.futures, argparse, and logging

No third-party dependencies are required for core functionality.

---

Installation

Follow these steps to install deduper for development or general use.

1. Clone Repository | Clone the repository using `git clone https://github.com/mmpouya/deduper.git`
2. Navigate Directory | Change to the cloned directory: `cd deduper`
3. Optional: Install as Package | Install locally with `pip install .` if you want to use it as a library
4. Ready to Use | Run deduper via Python or use its command-line interface

Tip

No external packages are needed. You can use deduper directly from source after cloning.

---

Features

• Recursive directory scanning for files
• Fast and configurable hashing for file comparison
• Multithreaded scanning and hashing for speed
• Flexible duplicate reporting and deletion options
• Command-line interface for automation and scripting
• Modular Python API for custom integrations
• Logging for progress and error reporting

---

Usage

Command-Line Interface

You can use deduper directly from the terminal. The main script provides options to scan directories, list duplicates, and remove them.

python deduper.py --path /your/target/directory --delete

Common Arguments:

• --path or -p: Path to the directory to scan (required)
• --delete or -d: Delete duplicate files (optional)
• --report-only or -r: Only report duplicates without deleting (optional)
• --threads or -t: Number of threads for scanning and hashing (optional, default: system CPU count)
• --hash or -h: Hash algorithm (optional, default: sha256)

Example: Reporting Duplicates

python deduper.py --path ./test_data --report-only

Example: Removing Duplicates

python deduper.py --path ./downloads --delete

Programmatic API

You can use deduper as a Python library to build custom workflows.

from deduper import DuplicateFinder

finder = DuplicateFinder(path="your/dir", hash_algo="sha256", num_threads=8)
duplicates = finder.find_duplicates()
finder.delete_duplicates(duplicates)

---

Configuration

You can customize deduper's behavior via command-line arguments or API parameters:

• path: Directory to scan (required)
• hash_algo: Hashing algorithm (e.g., 'sha256', 'md5')
• num_threads: Thread count for parallel processing
• delete: If True, delete duplicates; otherwise, only report
• report_only: If True, do not perform deletion

Note

The default hash algorithm is sha256 for a balance of speed and collision resistance.

---

Architecture Overview

The following diagram illustrates the core components and data flow within deduper:

---

Contact

For questions or feedback, please open an issue on the GitHub repository.

---

Important

Always back up your data before running deduper with the --delete option. The deletion of duplicates is irreversible.