Tech, ExplainedData platforms and AI
Product

Data catalog

Which table is the real customer list?

BEFORE ITAFTER ITData catalogAlationDataHubInformaticaCOVERSSearchLineageOwnershipAI descriptionsData warehouselistsits tablesData observabilityreportstable healthData security posturemanagement (DSPM)↗ ENTERPRISE CYBERSECURITYfindspersonal dataexplainseach columnAsking the datain plain language↗ BUSINESS INTELLIGENCEtagspersonal dataPrivacy management↗ ENTERPRISE CYBERSECURITYCONCEPTS IT USESModern data stackBEFORE ITAFTER ITData catalogAlationDataHubInformaticaCOVERSSearchLineageOwnershipAIdescriptionsData warehouselistsits tablesDataobservabilityreportstable healthData securityposturemanagement (DSPM)↗ ENTERPRISECYBERSECURITYfindspersonal dataexplainseach columnAsking the datain plain language↗ BUSINESSINTELLIGENCEtagspersonal dataPrivacymanagement↗ ENTERPRISECYBERSECURITYCONCEPTS IT USESModern data stackBEFORE ITData warehouse↓ lists its tablesData observability↓ reports table healthData security posture management (DSPM)↗ ENTERPRISE CYBERSECURITY↓ finds personal dataData catalogAlationDataHubInformaticaSearchLineageOwnershipAI descriptionsAFTER IT↓ explains each columnAsking the data in plain language↗ BUSINESS INTELLIGENCE↓ tags personal dataPrivacy management↗ ENTERPRISE CYBERSECURITYCONCEPTS IT USESModern data stack
The data catalog reads the list of tables from the data warehouse and health warnings from data observability, and sends column meanings to plain-language analytics and personal-data tags to privacy management.
opens a page inside this one a page in the same scene a page in another scene a concept page a label only, with no page

A data catalog is a searchable list of every table and report a company has, showing who owns each one, what each column means and where its data came from. It is also spelt data catalogue. A data team uses it the way a library uses its index: to find the right data quickly, and to know whether it may be used.

Why is it needed? After a few years, a company's data warehouse can hold tens of thousands of tables. Many have similar names, such as "orders", "orders_final" and "orders_v2". A new analyst cannot tell which one the finance team trusts. Without a catalog, she asks around on chat, and she may pick the wrong table.

Most of a catalog fills itself in automatically. It connects to the warehouse, the pipelines and the dashboards, and reads their metadata, the data about the data: table names, column names, data types and row counts. It also records lineage, the path that data takes from its source system, through each pipeline, into each table and each dashboard. People add what software cannot know: a plain description of each column, the name of the owner, and a "certified" tag on the tables the team trusts. More and more, an AI model reads the column names and sample values and drafts the descriptions, and an employee then checks them.

Consider a new analyst at a two-wheeler maker in Chennai. She must find last quarter's warranty claims for each dealer. She types "warranty claims" into the catalog, and it shows three tables. One is tagged certified. It is owned by an employee in the after-sales team, and it is refreshed every day at 6 a.m. Its lineage shows that it comes from the dealers' service system and feeds two dashboards. One of the other two tables has not been updated for eight months. She uses the certified table, and the task takes ten minutes instead of two days.

A catalog also helps protect privacy. It tags the columns that hold personal data, such as phone numbers and Aadhaar numbers. Data security posture management (DSPM) tools scan the company's systems to find such columns. Privacy management software then uses the tags to meet the company's duties under India's DPDP Act, for example when a customer asks for her data to be deleted.

The catalog's descriptions also help plain-language analytics. An AI model that writes a query for a question needs to know which column means what. And the catalog shows warnings from data observability next to each table, so an analyst can see that a table is late before she uses it.

Some catalogs are open source, and companies run them free of charge. Others are sold on their own, as part of a modern data stack, or inside a larger suite.

Tech, Explained is a part of ingrid.fyi. Every page explains one piece of technology in plain words.