Cross-platform entity reconciliation pipeline built on your data — feedback welcome

[805Burner66]

Hi @rhowardstone!

I'm a data analyst who's been building a cross-platform analytical reconciliation layer on top of your dataset, epstein-docs, and doc-explorer.

The project is called ECARE (EFTA Cross-Platform Analytical Reconciliation Engine). It produces a unified SQLite database and reproducible Python pipeline that:

• Resolves entities across all three platforms into a single canonical registry (18,800+ entities, using your persons_registry.json as the canonical base)
• Merges relationship graphs with full provenance tracking; every relationship traces back to which source systems asserted it
• Runs cross-source analytics: corroboration scoring, structural gap analysis, document coverage, temporal anomalies, contradiction detection

Some findings that might interest you:

• 3,093 relationships are now confirmed by 2+ independent sources. Your curated knowledge graph + doc-explorer RDF triples + epstein-docs co-occurrence data reinforce each other more than I expected.
• Lesley Groff appears in 177,773 corpus documents but has only 73 graph relationships, which is the single largest document-coverage gap in the dataset.
• Lefkowitz ↔ Acosta communication is confirmed by 3 independent sources, surfacing the plea-deal communication channel.
• Rodriguez ↔ Virginia Giuffre share 17 mutual connections but have no documented direct relationship; a structural gap worth investigating.

What I'd like from you:

1. Feedback on the entity resolution approach (I used your registry as the base, exact → alias → fuzzy matching at threshold 90)
2. Whether the unified database format would be useful to you or others in the community
3. Any known issues with the data I should account for (I documented what I found in methodology.md)

The repo is at [https://github.com/805Burner66/ECARE]. Everything runs with python run_pipeline.py and produces a single ecare.db file anyone can query. MIT licensed.

I deliberately built this as a data product, not a platform. The goal is to plug into existing projects like yours rather than compete with them.

Happy to discuss any of the methodology decisions. Thanks for the foundational work on the EFTA standard and knowledge graph, this project wouldn't exist without it.

[rhowardstone]

Oh wow! This is wonderful! I will take a closer look tomorrow but I could definitely see incorporating the tables to enrich my entity tables, if you're open to that

[rhowardstone]

Thanks again for sharing this, really impressive work! The corroboration scoring may actually server more purposes than you think! A few thoughts now that I'm awake and have had coffee:

• There are loads of additional data sources that could be cross-referenced. I have a directory of community platforms, and there's also the r/Epstein megathread tracking. For epstein-exposed, someone built an unofficial API wrapper that may prove useful.

• On incorporating the tables into my site, I'd like to walk back my enthusiasm slightly: my data has been through extensive PII redaction (~2,000 victim identifiers scrubbed), and I'd need to be very careful about importing unified entities from sources that may not have gone through such extensive screening. I think ECARE is most valuable as exactly what you described: a cross-reference layer that sits on top of (or, between) existing projects, rather than feeding back into them directly.

• One thing I'd there's a general concern in the community about data provenance and trust: people are asking how they can know a derivative dataset hasn't been sanitized, poisoned, or had inconvenient connections quietly removed. Your provenance tracking is a great start for addressing this. Have you thought about using ECARE to conduct such analyses? If we could verify that all information present in one dataset is also present in the others, that would go a long way toward building trust between platforms. Similarly, if there are systematic gaps between data sources, that itself could be an interesting signal to conduct further research. One might ask, "Why would someone not want this information searchable?"

Is there anything I can do on my end to help, re the 'blocking items'?

EDIT: I've just run across a lovely complementary project at: https://www.reddit.com/r/Epstein/comments/1rhoxzy/mass_file_changes_detected_in_doj_epstein/

Yet more sources to cross-reference: https://epsteinforums.org/viewtopic.php?t=20

[805Burner66]

Really appreciate the detailed response.

Your third point is the big one. I built this thinking about research gaps, but you're right that the same infrastructure works as an integrity verification system. If something gets quietly removed from one source, ECARE flags it because the other sources still have it. That's a fundamentally different value proposition than what I originally scoped.

I'm going to grab that mass file changes thread and use the flagged EFTA numbers as a test case. If the pipeline reliably catches known alterations, that's proof of concept.

Fully agree on PII, your redaction work is extensive and I don't want to compromise it. Happy to document my entity resolution methodology in detail so you can evaluate what's safe to cross-reference without risk.

Which additional sources would you prioritize next? And is there anything specific on your blocking items where I can help?

EDIT: Looked into this further, EpsteinExposed already has file-level integrity covered with rodrigopolo's SHA-256 hashes and a live monitoring dashboard. But I think what you're describing is a layer above that: verifying that derivative analyses across platforms haven't had connections selectively omitted, whether by intent or oversight. That's a different problem, rather than 'did this PDF change on the DOJ server' but 'does a relationship that exists in your knowledge graph also show up in other platforms' analyses, and if not, why not.' ECARE's cross-source reconciliation is already set up to answer that. I'll scope out what a systematic version looks like.

[rhowardstone]

I've been pulled in quite a few directions at once. I am curious about what gaps in just files alone exist -- a cross-platform reconciliation of which EFTA ('Bates') numbers are represented between the various sites would be super interesting towards weeding out any of.. suspect origin.. shall we say. I ran a similar audit at:

https://github.com/rhowardstone/Epstein-research-data/blob/main/doj_audit/CONFIRMED_REMOVED.csv

I have accounted for 100% of EFTA numbers from EFTA00000001 up to EFTA02858498 (DS12 just dropped 23 new files) - so, if you simply query which files people claim to have, it might save you the trouble of measuring the hash values to determine if they are different (although of course, that would also be a fascinating analysis)

EDIT: there is currently a gap between EFTA02825344 (https://epstein-data.com/EFTA02825196#page=149) and EFTA02829305 (https://epstein-data.com/EFTA02829305) - but these records may exist elsewhere on https://www.justice.gov/epstein/doj-disclosures. The larger gap, created by the release of the 23 new files, from the end of the "dataset 12" files EFTA02731852 (https://www.justice.gov/epstein/files/DataSet%2012/EFTA02731852.pdf) up to EFTA02857524 (https://www.justice.gov/epstein/files/DataSet%2012/EFTA02857524.pdf), was filled in with the maxwell cases on the doj-disclosures page

Edit2: The gap referenced above can be filled by the Maxwell defense docs. For URLs, please see epstein-data.com/EFTAxxxxxxxx