Tech, ExplainedData platforms and AI
Product

Data warehouse

Every transaction, copied in to be questioned.

BEFORE ITAFTER ITData warehouseTeradataSnowflakeCOVERSColumn storageSQL queriesPay per queryKept apartETL and ELTloads eachnight's dataChange data capture (CDC)copieseach changesupplieschart figuresBusiness intelligence(BI) dashboards↗ BUSINESS INTELLIGENCEanswers typedquestionsAsking the datain plain language↗ BUSINESS INTELLIGENCEsuppliestrainingtablesAutomated machinelearning (AutoML)↗ GENERATIVE AI APPLICATIONSCONCEPTS IT USESOLTP and OLAPColumnar databaseVendor lock-inBEFORE ITAFTER ITData warehouseTeradataSnowflakeCOVERSColumn storageSQL queriesPay per queryKept apartETL and ELTloads eachnight's dataChange datacapture (CDC)copieseach changesupplieschart figuresBusinessintelligence(BI) dashboards↗ BUSINESSINTELLIGENCEanswers typedquestionsAsking the datain plain language↗ BUSINESSINTELLIGENCEsuppliestrainingtablesAutomated machinelearning (AutoML)↗ GENERATIVEAI APPLICATIONSCONCEPTS IT USESOLTP and OLAPColumnar databaseVendor lock-inBEFORE ITETL and ELT↓ loads each night's dataChange data capture (CDC)↓ copies each changeData warehouseTeradataSnowflakeColumn storageSQL queriesPay per queryKept apartAFTER IT↓ supplies chart figuresBusiness intelligence (BI) dashboards↗ BUSINESS INTELLIGENCE↓ answers typed questionsAsking the data in plain language↗ BUSINESS INTELLIGENCE↓ supplies training tablesAutomated machine learning (AutoML)↗ GENERATIVE AI APPLICATIONSCONCEPTS IT USESOLTP and OLAPColumnar databaseVendor lock-in
The data warehouse receives each night's records from ETL and ELT pipelines and each change from change data capture, and answers the questions asked by BI dashboards and plain-language analytics tools.
opens a page inside this one a page in the same scene a page in another scene a concept page a label only, with no page

A data warehouse is a separate database into which a company copies the records of all its other systems, so that analysts can ask questions across all of them at once. Banks, retailers and app companies all use one. It is also called an enterprise data warehouse, or EDW. Today, most companies rent a cloud data warehouse, instead of buying their own servers for it.

A warehouse is different from the database an app runs on. The app's database, such as a relational database, records each order the moment it happens. It is built for many small writes. A warehouse is built for a few very large reads: questions that add up millions of rows. These two kinds of work are explained in OLTP and OLAP. Keeping them apart means that a heavy question never slows down the customers using the app.

Most warehouses store data column by column, as a columnar database does. A query that adds up the price of every sale reads only the price column, not every detail of every sale. A column also holds similar values, so it compresses well and takes less disk space.

Data reaches the warehouse in two ways. An ELT pipeline copies data from each source system on a schedule, often every night. Change data capture (CDC) copies each change within seconds of it happening.

Take, for example, a clothing retailer with 300 stores across south India, run from Coimbatore. Its billing system, its online shop and its returns desk each keep their own records. Every night, the day's 1.5 lakh sales and 4,000 returns are copied into the warehouse. During Pongal week, the head of merchandising asks which silk saree designs sold best in Madurai, and how many of them came back. The query joins sales and returns from three systems and reads 90 lakh rows. It returns in about eight seconds. The answer then appears on the company's business intelligence (BI) dashboards.

How is a cloud warehouse paid for? Usually, storage and computing are charged separately. The company pays a small monthly amount for every terabyte it stores. It pays separately for the computing that each query uses. So a badly written query that reads the whole history every few minutes shows up on the bill. Data observability tools now look for such queries.

The choice of warehouse is hard to reverse. Years of tables, queries and dashboards get built around one product. This is called vendor lock-in. A data lakehouse tries to reduce it, by keeping the data in open file formats that other software can also read.

Tech, Explained is a part of ingrid.fyi. Every page explains one piece of technology in plain words.