An unattended B2B lead pipeline with staged execution, cross-pipeline dedupe, and a control plane that refuses to run on a failed day.
Three parts, ~4,400 lines of Python. Published because the failure analysis is more useful than the engine.
Over its run, this system contacted 227 businesses and tracked zero replies.
That is the honest headline, and it is corroborated inside the system's own data: the outcomes tracker finished with every row still marked NOT CONTACTED, because outcome logging was never wired to the send path. The engine was excellent at finding, scoring, formatting and sending. It had no idea whether any of it worked.
If you take one thing from this repository, take that: a lead engine without a reply path is a lead engine that cannot learn. Volume was never the constraint. Measurement was.
volume/ |
The daily pipeline. Rotates a different slice of verticals each day, crawls a maps source, enriches, scores, dedupes, formats and sends. |
signal/ |
The narrow counterpart. Sweeps a few capability-led verticals per day, indexed off day-of-year so the mix rotates, and scores on fit rather than volume. |
control/ |
The control plane. Owns the day's state and refuses to advance a stage when the previous one failed. |
The guardrails fired, and correctly. One volume run died mid-way in merge_score.py. The control plane caught it, and the signal pipeline self-paused rather than running on a half-built day. Nothing was sent from a broken state. That is the behaviour you want and it is the hardest part to test, because it only proves itself on a bad day.
Dedupe by inbox, not by row. The send guard reads an append-only log, so an interrupted run never re-sends. But that only protects across runs — it cannot see a duplicate sitting inside a single pool. In one real run, six emailable rows resolved to only four inboxes: one business was listed twice on the maps source (the practitioner and the centre at one address), and another inbox sat centrally behind two branch listings.
Because the message body never contains the business name, the second copy is byte-identical to the first — which proves the opening line "I had a look at your…" was not true. One inbox, one email. Collapse the pool before sending, not after.
Treat a set finishedAt as terminal. The crawl poller had four runs that were genuinely finished report a non-terminal status for the full timeout window. Polling on status alone discarded completed, paid-for datasets. The fix is to trust the timestamp: if the run has a finish time, it is finished, whatever the status field says.
- Stages pin a hardcoded date. Several modules default to one specific day's directory rather than deriving it. Runs on any other day silently fall back instead of failing loudly, which is the worst of both.
- No outcome loop. As above. Sends are logged; replies are not.
- Duplicate engine snapshots existed in the source tree from recovery runs and are excluded here.
They are listed because a repository that only shows the parts that worked is a brochure.
export LEAD_ENGINE_BASE=~/lead-engine-data/volume
export LEAD_ENGINE_SIGNAL=~/lead-engine-data/signal
export CAMPAIGN_FROM="you@yourdomain.com"
python3 volume/_engine_run_daily.py # dry run by default
python3 signal/_engine_run_signal.pySending is dry-run unless you pass --send. That default is deliberate and should stay that way.
Not one row of real contact data is in this repository, and none is recoverable from its history. The source trees held roughly 300 third-party email addresses and 1,300 phone numbers belonging to businesses that never consented to collection; the outbound ledger recorded who was contacted and with what. None of it is here, .gitignore blocks every format it lived in, and the two third-party inboxes that appeared in a code comment have been replaced with a description of the pattern.
If you run this: contact data is personal data. Applicable law, platform terms and anti-spam rules are yours to satisfy, and a send list is the single most sensitive artefact the system produces.
MIT licensed.