Tech, ExplainedData platforms and AI
Product

Data quality and enrichment

One Pune flat, written forty different ways.

BEFORE ITAFTER ITData qualityand enrichmentPreciselyInformaticaCOVERSAddress parsingStandardisingValidationEnrichmentETL and ELTpassesnew recordsDigital maps andlocation data↗ GEOSPATIAL DATAaddsmap locationssends cleanedrecordsMaster datamanagement (MDM)BEFORE ITAFTER ITData qualityand enrichmentPreciselyInformaticaCOVERSAddressparsingStandardisingValidationEnrichmentETL and ELTpassesnew recordsDigital maps andlocation data↗ GEOSPATIAL DATAaddsmap locationssends cleanedrecordsMaster datamanagement (MDM)BEFORE ITETL and ELT↓ passes new recordsDigital maps and location data↗ GEOSPATIAL DATA↓ adds map locationsData quality and enrichmentPreciselyInformaticaAddress parsingStandardisingValidationEnrichmentAFTER IT↓ sends cleaned recordsMaster data management (MDM)
Data quality and enrichment receives new records from the ELT pipeline and map locations from digital maps, and sends cleaned, complete records to master data management.
opens a page inside this one a page in the same scene a page in another scene a concept page a label only, with no page

Data quality and enrichment software finds wrong, missing and inconsistent details in a company's records, corrects them, and fills in missing details from trusted sources. Data quality means checking and fixing records. Enrichment means adding details that a record lacks, such as a PIN code, a district or a location on a map. Together, they make records reliable enough to deliver to, to bill and to lend against.

Addresses show why this is hard. The same flat in Pune can be written as "Flat 302, Shanti Apts, Kothrud", as "302 Shanti Apartment, near Karve Road, Pune 411038", or as "B-302 Shanti Apartments Kothrud". A program that compares the text letter by letter treats these as three different addresses.

The software works in steps. First, it parses each address: it splits it into parts such as flat number, building, street, locality, city and PIN code. Second, it standardises each part, for example by turning "Apts" and "Apartment" into one form. Third, it validates the parts against reference data, such as the official list of PIN codes and the localities in each one. Finally, it enriches the record, for example by adding the latitude and longitude from a digital map. The same steps apply to names, phone numbers, GSTINs and email addresses.

Imagine a courier company that delivers 30,000 parcels a day in Pune. About a tenth of its addresses arrive without a PIN code, or with a PIN code that does not match the locality. A parcel with a bad address goes to the wrong sorting hub, and it loses a day and needs a second trip. So the company runs every new address through data quality software before the parcel is sorted. The software adds the missing PIN codes, corrects the wrong ones and adds the map location. About 400 addresses a day are too unclear to fix automatically. Those go to an employee in the customer service team, who calls the customer.

Quality checks can run inside an ELT pipeline, as data is loaded into the warehouse. They can also run inside a business system, as each record is entered. They are the first step of master data management, which matches records only after each record is clean. Some versions are built for one kind of record, such as provider data management, which cleans the details of doctors for a health insurer.

How is this different from data observability? Observability watches whole tables: did they arrive on time and in full? Data quality looks inside each record: is this address real and complete?

Tech, Explained is a part of ingrid.fyi. Every page explains one piece of technology in plain words.