Tech, ExplainedData platforms and AI
Part

Data transformation

Raw tables cleaned and joined inside the warehouse.

INSIDE: ETL AND ELTHOW IT WORKS · CLEANING A BICYCLE MAKER'S SALES TABLESRaw dealer orders,online ordersand returns landin the warehouseModel 1 fixes thedates and removes312 duplicatedealer ordersModel 2 converts4,59,900 paise to₹4,599 for eachonline orderModel 3 joins salesand returns: 2,140sold in Punjab inOctober, 96 returnedA test checks thatno order appearstwice before thedashboard updatesINSIDE: ETL AND ELTHOW IT WORKS · CLEANING ABICYCLE MAKER'S SALES TABLESRaw dealer orders, onlineorders and returns landin the warehouseModel 1 fixes the datesand removes 312 duplicatedealer ordersModel 2 converts 4,59,900paise to ₹4,599 for eachonline orderModel 3 joins salesand returns: 2,140sold in Punjab inOctober, 96 returnedA test checks thatno order appearstwice before thedashboard updatesINSIDE: ETL AND ELT · CLEANING A BICYCLEMAKER'S SALES TABLESRaw dealer orders, online orders andreturns land in the warehouse↓Model 1 fixes the dates and removes 312duplicate dealer orders↓Model 2 converts 4,59,900 paise to ₹4,599for each online order↓Model 3 joins sales and returns: 2,140sold in Punjab in October, 96 returned↓A test checks that no order appears twicebefore the dashboard updates
Data transformation turns a bicycle maker's raw dealer, online and returns tables into one clean table of daily sales by state, removing 312 duplicates and converting paise to rupees on the way.
opens a page inside this one a page in the same scene a page in another scene a concept page a label only, with no page

Data transformation is the step that turns raw tables into clean tables that people can trust and use. It cleans the data, fixes formats, removes duplicates and joins tables together. In ELT, it is the T, and it runs inside the warehouse after the raw data has been loaded by ETL and ELT tools.

Raw data arrives exactly as each source system stores it. One system writes dates as 03/11/2026, and another as 2026-11-03. One stores money in rupees, and another in paise. A customer may appear twice because a form was submitted twice. If an analyst adds up raw tables like these, the totals are wrong.

Transformations are usually written in SQL. Each one is a short query, often called a model, that reads one or more tables and produces a new, cleaner table. The models are saved as files and kept in version control, like any other code, so that every change is recorded. A tool runs them in the right order, because a later model depends on earlier ones. The tool also runs tests on the results, such as "no order number appears twice", and stores a description of every table and column.

Consider a bicycle maker in Ludhiana that sells through dealers and through its own website. Its raw tables come from the dealer ordering app, the online store and a returns sheet. The first model cleans the dealer orders. It converts the dates to one format, and removes 312 duplicate rows that appeared when dealers pressed Submit twice. The second model converts the online store's amounts from paise to rupees, so that 4,59,900 paise becomes ₹4,599. The third model joins both with the returns, and produces a table of daily sales by state. It shows 2,140 bicycles sold in Punjab in October, of which 96 came back. One morning, the duplicate test fails, because a new kind of duplicate has appeared. The tool stops, and the sales dashboard keeps yesterday's correct numbers instead of showing wrong ones.

The clean tables feed the business intelligence dashboards that managers read. The descriptions of each table appear in the data catalog. And when a test fails, data observability tools help find out why.

Transformation used to take much of a data engineer's day. Now AI tools write many of the models from a description, run them, read the errors and fix them, which the page on AI coding agents explains. Some also write the documentation. A related idea, the AI data analyst, writes queries to answer a manager's question directly.

In the older ETL order, the same cleaning happened before the data reached the warehouse, on a separate server. Doing it inside the warehouse means the raw data is kept, so a wrong model can be fixed and run again.

Tech, Explained is a part of ingrid.fyi. Every page explains one piece of technology in plain words.