prospect-dedup-engine

A Python CLI tool that deduplicates B2B lead lists by email, company domain, and phone number — handling variations, typos, and formatting differences.

What It Does

Raw CRM exports and lead lists almost always contain duplicates: the same contact entered twice with a slightly different email, a company name that varies between "Acme Inc" and "Acme Limited", or a phone number formatted differently across systems. This tool catches all of those cases in a single pass.

Output: a clean deduplicated CSV plus a conflict report showing every pair that was merged and why.

---

Deduplication Logic

Three passes run in order. Earlier passes take priority.

Pass	Signal	Detail
1	Exact email	Case-insensitive exact match on the email field
2	Domain + fuzzy company name	Same email domain AND company names ≥ 85% similar (difflib SequenceMatcher) — catches james@acme.com vs j.omondi@acme.com when both list "Acme Inc" vs "Acme"
3	Normalised phone	Strips all non-digits, compares last 10 digits — +254712345678, 0712345678, and 712345678 all match

Master record selection: within each duplicate group, the record with the most non-empty fields is kept as the MASTER. All others are removed from the clean list and logged in the report.

---

Installation

pip install -r requirements.txt

Requires Python 3.9+.

---

CLI Usage

python dedup.py --input leads.csv --output clean_leads.csv --report report.csv

Flag	Description
--input	Path to the raw CSV file (must include headers)
--output	Path for the deduplicated output CSV
--report	Path for the conflict report CSV

Required CSV columns: email, company, phone. Any additional columns are preserved.

---

Sample Output

Console summary

=============================================
  Deduplication Summary
=============================================
  Total input records   : 20
  Duplicates removed    : 7
  Output records        : 13
  Reduction             : 35.0%
=============================================
  Clean list  -> clean_leads.csv
  Report      -> report.csv
=============================================

report.csv (excerpt)

master_id	master_email	master_company	duplicate_id	duplicate_email	duplicate_company	reason
1	james.omondi@safaricom.co.ke	Safaricom Ltd	3	james.omondi@safaricom.co.ke	Safaricom	EMAIL
2	amina.waweru@equity.co.ke	Equity Bank Kenya	7	awaweru@equity.co.ke	Equity Bank	DOMAIN
4	b.njoroge@kcb.co.ke	KCB Group	10	brian.njoroge@kcb.co.ke	KCB Group Ltd	PHONE

---

Project Structure

prospect-dedup-engine/
├── dedup.py          # CLI entrypoint
├── utils.py          # normalize_phone(), extract_domain(), fuzzy_name_match()
├── sample_leads.csv  # 20 realistic B2B leads with intentional duplicates
├── requirements.txt
└── README.md

---

Run Against the Sample Data

python dedup.py --input sample_leads.csv --output clean_leads.csv --report report.csv

The sample file contains 20 rows with 7 embedded duplicates across all three dedup types (EMAIL, DOMAIN, PHONE).

---

Part of the Datastag GTM Toolchain

Built for Datastag — data infrastructure for growth teams.