3f:null 40:["$","title",null,{"children":"Entity Resolution At Scale | Nube Technologies | Observable"}] 46:[] 44:{"id":"fdc34b0b0137c26c","slug":"entity-resolution-at-scale","title":"Entity Resolution At Scale","thumbnail":"54ced672de4d740efd2c56b591d3c9ce514fd88ed1d093a1ddb295d42f4b283c","access_level":"public","create_time":"$D2020-02-14T07:31:29.788Z","update_time":"$D2020-02-16T08:08:56.621Z","view_time":null,"first_public_time":"$D2020-02-14T16:29:37.375Z","has_unpublished_edits":null,"fork":{"id":"71cf6720f671feea","slug":"bubble-o-matic","title":"Bubble-o-Matic","owner":{"id":"8c2e47252fd4995c","login":"mbostock","name":"Mike Bostock"}},"stars":4,"starred":false,"owner":{"type":"team","id":"8a27a50e629aa5b6","login":"nubetech","name":"Nube Technologies","avatar_url":"https://avatars-next.observableusercontent.com/370c8613ed2cc46d36272e3180fa328e30281c71fdf21b0558236604e814282e","tier":"starter_2024","bio":"AI for MDM, Data Quality, KYC, AML","following":false},"version":551,"publish_version":551,"role":null,"authored":false,"authors":[{"type":"individual","id":"28188a6a08381615","login":"nubetech","name":"Nube Technologies","avatar_url":"https://avatars-next.observableusercontent.com/370c8613ed2cc46d36272e3180fa328e30281c71fdf21b0558236604e814282e","tier":null,"bio":"AI for MDM, Data Quality, KYC, AML","following":false}],"members":[{"type":"individual","id":"28188a6a08381615","login":"nubetech","name":"Nube Technologies","avatar_url":"https://avatars-next.observableusercontent.com/370c8613ed2cc46d36272e3180fa328e30281c71fdf21b0558236604e814282e","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":2,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Entity Resolution At Scale","cells":[{"id":0,"value":"md`# Entity Resolution At Scale`\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":444,"value":"html`Entity Resolution is the process of finding and linking records of an entity when a common identifier like customer id does not exist. An entity could be any business object - customer, vendor, product, location, event. Across an enterprise, multiple databases, applications and other systems may have variations in this entity.` ","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":463,"value":"html `Let us take customer journey in insurance as an example. At the very beginning, a customer is represented as a lead in the CRM. If the marketing channel is purely digital, the email will be valid, but other attributes like phone number may not be present. The name could be saved as firstName, lastName with address as one single field. As the customer buys a general insurance, the application database for general insurance gets populated with more precise attributes like father's name, date of birth etc. The name and address could be split up as per application database design, most likely first name, middle name, last name and address1, address2, city, state, country and zipcode. At some point in time, the customer buys a motor insurance. Now, the motor insurance application could save customer details as salutation + full name, address, city, state, country. This results in a customer represented in multiple ways across various enterprise systems.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":467,"value":"html `Data spread across different systems makes enterprise analysis impossible for customer lifetime value, cross selling, upselling, householding, fraud, risk etc. Sales, personalized and targetted marketing as well as compliance suffer. Lack of accurate and consistent customer data hampers machine learning models and predictive analytics. So we want to unify and join these datasets to build a holistic view of the customer.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":475,"value":"html `However, typos, blanks, salutations, different formats for phone numbers and addresses, abbreviations, split database fields, out of order components and other variations make joining dificult.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":469,"value":"html `As there is no unique identifier, we are essentially comparing every record with every record, which becomes computationally prohibitively expensive. So for a dataset of 1 million records, we need to compare each record with 1 million records which is 100,00,00,00,00,00 comparisons`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":447,"value":"html `

Here is how the comparisons blow up as the data size increases

`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":325,"value":"{\n return svg``;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":453,"value":"html`We can argue that parallelization can solve this, but Entity Resolution by its very nature does not scale linearly. Even a 1m dataset needs 100,00,00,00,00,00,00 comparisons. So, Entity Resolution is challenging for common sized datasets. Parallelization of cartesian joins is horribly difficult. Also, as the above chart shows, going from 1m to 10m records is not merely 10 times more compute. So beyond the issue with formulating the rules for entity matching, Entity Resolution is also a scale problem.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":14,"value":"md`---\n\n## Appendix`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":276,"value":"viewof source = {\n const textarea = html`

\n10m,10000000,#f64b3c\n5m,5000000,#516f88\n1m,1000000,#24ceb4`;\n  textarea.style.display = \"block\";\n  textarea.style.boxSizing = \"border-box\";\n  textarea.style.width = \"calc(100% + 28px)\";\n  textarea.style.font = \"var(--mono_fonts)\";\n  textarea.style.minHeight = \"60px\";\n  textarea.style.border = \"none\";\n  textarea.style.padding = \"4px 10px\";\n  textarea.style.margin = \"0 -14px\";\n  textarea.style.background = \"rgb(247,247,249)\";\n  textarea.style.tabSize = 2;\n  textarea.oninput = () => {\n    textarea.style.height = \"initial\";\n    textarea.style.height = `${textarea.scrollHeight}px`;\n  };\n  yield textarea;\n  textarea.oninput();\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":16,"value":"data = {\n  const data = d3.csvParseRows(source, ([name, value, color]) => ({name, value: !value || isNaN(value = +value) ? undefined : value, color}));\n  return data;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":289,"value":"erScale = d3\n  .scaleLinear() // instead of scaleLinear()\n  .domain([0, data[0].value])\n  .range([0, 200])","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":25,"value":"width = 1254","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":28,"value":"height = 720","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":539,"value":"index = 0","pinned":true,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":21,"value":"d3 = require(\"d3@5\")","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","tool":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W46","stdlib":"1"}}
45:"$44:body"
</article></body></html>