Tech, ExplainedData platforms and AI
Product

Data lakehouse

Lake prices with warehouse tables.

BEFORE ITAFTER ITData lakehouseDatabricksClouderaTeradataPARTOpen table formats+COVERSOpen formatsCheap storageSQL tablesAI trainingETL and ELTloads batchesof recordsEvent streamingaddslive eventssuppliestraining dataDistributedcomputing framework↗ GENERATIVE AI APPLICATIONSsharescleanedtablesData warehouseCONCEPTS IT USESObject storageColumnar databaseVendor lock-inBEFORE ITAFTER ITData lakehouseDatabricksClouderaTeradataPARTOpen table formats+COVERSOpen formatsCheap storageSQL tablesAI trainingETL and ELTloads batchesof recordsEvent streamingaddslive eventssuppliestraining dataDistributedcomputingframework↗ GENERATIVEAI APPLICATIONSsharescleanedtablesData warehouseCONCEPTS IT USESObject storageColumnar databaseVendor lock-inBEFORE ITETL and ELT↓ loads batches of recordsEvent streaming↓ adds live eventsData lakehouseDatabricksClouderaTeradataOpen table formats+Open formatsCheap storageSQL tablesAI trainingAFTER IT↓ supplies training dataDistributed computing framework↗ GENERATIVE AI APPLICATIONS↓ shares cleaned tablesData warehouseCONCEPTS IT USESObject storageColumnar databaseVendor lock-in
The data lakehouse receives batches from ETL and ELT tools and live events from event streaming, keeps one copy as tables over files, and supplies the same files to model training and to the data warehouse.
opens a page inside this one a page in the same scene a page in another scene a concept page a label only, with no page

A data lakehouse is one system that keeps a company's data as cheap files in a data lake, and also lets analysts question those files as clean tables, as they would in a data warehouse. The name joins the two words. Companies also use the same files to train AI models.

To see why it exists, start with its two halves. A data lake is a large store of raw files of any kind: app logs, sensor readings, images and exported tables. It usually sits in object storage, the cheapest way to keep huge amounts of data in the cloud. But a lake has no rules. Nobody checks that today's files match yesterday's. And if one program is writing files while another is reading them, the reader can get a half-finished answer. A data warehouse has strict tables and reliable answers. But it costs more, and it keeps the data in its own internal format.

For years, many companies kept both. They copied data from the lake into the warehouse every night. That meant two copies, two bills, and warehouse data that was often a day old. A lakehouse keeps one copy, in the lake. It adds a layer of table information on top of the files, written in an open table format. This layer records which files make up each table, and which version of the table is current. So the files behave like database tables, and a query engine can read them with SQL, the standard language for asking questions of tables.

Consider a food delivery company in Gurugram. Its app records each rider's GPS position every few seconds, about 50 crore readings a day, written as files in the lake. Analysts use SQL on those files to find the restaurants where riders wait longest. The data science team uses the same files, without copying them, to train a model that predicts each order's delivery time. The training runs on a distributed computing framework across many machines. The finished model then goes to a model deployment platform, which runs it for every new order.

New data arrives in two ways. ETL and ELT tools load batches from the company's other systems, and event streaming adds the GPS readings as they arrive.

Why do buyers care? The data stays in open formats, in the company's own cloud storage. If the company changes its query engine later, it does not have to move the data. This reduces vendor lock-in, although the code built around one vendor's tools can still be hard to move. Lakehouses also suit fields with very large files, such as a healthcare data platform that holds scans and lab results.

Tech, Explained is a part of ingrid.fyi. Every page explains one piece of technology in plain words.