You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/zingg-concepts/zingg-entity-resolution-platform/the-problems-zingg-is-built-to-solve-and-why-they-are-harder-than-it-looks.md
+4-2Lines changed: 4 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,13 +10,15 @@ Entity resolution at scale is not a data quality problem. It is an algorithmic c
| **The N² comparison problem** | <p>Naive entity resolution compares every record against every other record. With N records, that produces N × (N-1) / 2 comparisons.</p><ul><li>At 10,000 records: ~50 million comparisons.</li><li>At 1 million records: ~500 billion comparisons.</li><li>At 10 million records: ~50 trillion comparisons.</li></ul><p>No rules engine can evaluate 50 trillion pairs. No team can review them. And no cloud compute budget can sustain running that comparison at production cadence - weekly, daily, or on arrival of new records.This is not a hardware problem. It is a structural one. The comparison space grows quadratically with dataset size, and linear scaling of compute cannot outrun quadratic growth.Zingg's blocking model is the solution. Before any similarity comparison runs, the blocking model groups records into candidate buckets using learned field heuristics. Only records within the same bucket are ever compared. Typical Zingg comparison coverage is 0.05% to 1% of the full problem space - without losing the matching pairs that matter.</p> |
12
12
| **The variation problem - why rules always break** | Entities do not appear consistently across systems. "IBM", "I.B.M.", and "International Business Machines" are the same company. "Jon Smith" and "Jonathan Smith" may be the same person. "42 Main St" and "42 Main Street, Apt 3B" are likely the same address.A rule-based approach requires you to enumerate every possible variation for each field. For names alone, the number of possible abbreviations, abbreviation types, and transliteration patterns is unbounded. For addresses, it is worse. For company names across geographies and time periods, it is not practically solvable with rules.Zingg's similarity model handles variation by computing multiple field-level features, character differences, string lengths, common transpositions, prefixes, suffix overlaps, and combining them through a classifier trained on your labeled pairs.The classifier learns the variation patterns in your specific data. It does not need you to enumerate them in advance. |
13
-
| **The confidence problem - why binary match/no-match is not enough** | <p>A rule either fires or it does not. It gives you a binary answer: match or no match.Production entity resolution requires more than that. Some clusters are high confidence - strong matches across multiple fields that can go straight to automated golden record creation. Others are lower confidence plausible matches that a domain expert should review before merging. Others are borderline pairs that look similar but are probably different entities.Without a graded confidence signal, every match decision has to be treated the same way. You either automate everything and accept errors or review everything manually and abandon the scale benefit entirely. Zingg's similarity model produces a score between 0 and 1 for every candidate pair. That score drives <code>Z_MINSCORE</code> and <code>Z_MAXSCORE</code> on every cluster in your output - so you can route high-confidence clusters to automated processing and low-confidence clusters to a stewardship queue.</p><p></p><p><strong>Read more</strong>:</p><ul><li><a href="../../interpreting-results/interpret-output-scores.md">Interpret Output Scores</a></li></ul> |
13
+
| **The confidence problem - why binary match/no-match is not enough** | <p>A rule either fires or it does not. It gives you a binary answer: match or no match. Production entity resolution requires more than that. Some clusters are high confidence - strong matches across multiple fields that can go straight to automated golden record creation. Others are lower confidence plausible matches that a domain expert should review before merging. Others are borderline pairs that look similar but are probably different entities.Without a graded confidence signal, every match decision has to be treated the same way. You either automate everything and accept errors or review everything manually and abandon the scale benefit entirely. Zingg's similarity model produces a score between 0 and 1 for every candidate pair. That score drives <code>Z_MINSCORE</code> and <code>Z_MAXSCORE</code> on every cluster in your output - so you can route high-confidence clusters to automated processing and low-confidence clusters to a stewardship queue.</p><p></p><p><strong>Read more</strong>:</p><ul><li><a href="../../interpreting-results/interpret-output-scores.md">Interpret Output Scores</a></li></ul> |
14
14
| **The pairs are not enough problem** | <p></p><p>After the similarity model scores every candidate pair, a graph algorithm groups them into complete clusters using transitive closure.</p><p>If Record A matches Record B, and Record B matches Record C, the algorithm concludes that A, B, and C all represent the same entity and groups them into a single cluster, even if A and C were never directly compared.</p><p>This is what turns a list of scored pairs into a usable identity graph. Every resolved entity becomes a node. Every cluster is an entity resolved across all its representations.</p><p>In Community, clusters are assigned a <code>Z Cluster</code> that is non-persistent between runs. In Enterprise, each cluster is assigned a persistent <code>Zingg ID</code> (GUID) that remains stable across runs, incremental updates, and model changes.</p><div data-gb-custom-block data-tag="hint" data-style="success" data-icon="right-long" class="hint hint-success"><p><strong>Read more</strong>:</p><ul><li><a href="../identity-graph.md">Identity Graph</a></li><li><a href="../z-cluster-and-zingg-id.md">Z Cluster ID vs Zingg ID</a></li><li><a href="/broken/pages/9QpDFW20AMt0UJ4cEW6b">Zingg ID</a></li></ul></div> |
15
15
| **Managing addition, updation and deletion of records** | <p>Production entity resolution rarely sees a static dataset. New records arrive daily; sometimes hourly. The naive approach is to rerun the full match each time new data comes in. This breaks at scale for two reasons. First, every re-run produces fresh cluster IDs. Downstream systems that store the cluster ID as a customer or entity key break every time the match is re-run.</p><p></p><p>Zingg Enterprise's <code>runIncremental</code> phase handles this. New records are evaluated against the existing identity graph. Records matching an existing cluster inherit its Zingg ID. Records that do not match any existing cluster receive a new Zingg ID. The existing graph stays intact. Downstream systems keep working.<br><br>Second, re-running the full match is computationally expensive and slows down your pipelines. Doing it every time a thousand new records arrive is not the best way to process data.<br><br><strong>Read more</strong>:</p><ul><li><a href="../../running-zingg/run-incremental-matching.md">Run Incremental Matching</a></li></ul> |
16
16
| **The data drift problem - why models decay** | <p>Data patterns change. New source systems are added. Naming conventions shift. A model trained on last year's data may perform poorly on this year's records, not because the model was wrong, but because the variation distribution in your data changed. <br><br></p><p>Most systems have no mechanism to detect or adapt to these changes. Zingg Enterprise's compare model results feature (<code>diff</code> phase) lets you benchmark a retrained model against your current production model before deploying. So, you can see exactly what changed before the change goes live. Zingg Enterprise also provides reassign feature, so you can continue with existing <code>ZINGG ID</code>s even while upgrading models.<strong>Read more</strong>:</p><ul><li><a href="../../running-zingg/compare-model-results.md">Compare Model Results</a></li><li><a href="../../running-zingg/reassign-zingg-id.md">Reassign Zingg ID</a></li></ul> |
17
17
| **The governance gap** | Explaining how records matched or why they did not match is critical for audit and compliance, espeically in regulated industries. Zingg Enterprise comes with a whole suit of explanations at the record and cluster level to help stakeholders understand the matching. |
18
18
| **The domain experience gap** | <p>Some records need special treatment. For example Beth, Eliza and Betty. Same person, different nicknames. Or IBM and International Business Machine. Zingg's mapping framework promotes domain expert supplied dictionaries to push matching accuracy further. <br><br>Sometimes records contain information that is not sharp enough for meaningful signals - like common words Pvt, Ltd etc. Zingg recommends and allows domain expert to plugin the relevant stopwords to ignore during matching. <br><br>Together with the SME labeling the edge cases, this provides unmatched matching accuracy(pun intended <span data-gb-custom-inline data-tag="emoji" data-code="1f604">😄</span>)</p> |
19
19
| **The schema problem** | <p>Since matching is tied heavily to fields, most tools end up imposing restrictions on attributes which can be present in the input. Zingg takes a completely different approach. By learning directly from the data, Zingg works with any fields and their combinations, with no hard requirements for mandatory email, phone number or other fields. </p><p></p><p>Zingg's approach also powers any entity type like supplier, product etc, not just customer. </p> |
20
-
| **The data privacy problem** | Entity resolution is typically performed on the most sensitive data woned by the Enterprise. Customer lists, Vendor Info, Leads. Sending such data to a third party or a public cloud service |
20
+
| **The data privacy problem** | Entity resolution is typically performed on the most sensitive data woned by the Enterprise. Customer lists, Vendor Info, Leads. Sending such data to a third party or a public cloud service is a major privacy and compliance nightmare. Through warehouse native entity resolution, Zingg ensures your data remains private. |
21
+
| **The data pipeline problem** | <p>Entity Resolution is a fundamental construct in data management and has to align with the enterprise's overall architecture needs. </p><p></p><p>Zingg's design in all editions ensures entity resolution can run part of your data proessing pipeline, at a cadence of your choice. <br><br>Zingg Enterprise provides batch and streaming flows to cater to different architecture choices and design patterns.<br><br>By running directly within the data pieline, Zingg maximises data infrastructure ROI since no extra ETL and observability needs to be set for Zingg. </p> |
0 commit comments