Data pipelines are the code that copies data out of every system where it is created, keeps the copy current, and cleans it on the way into a data warehouse. The software engineers who build and run pipelines are called data engineers. The data team is the main buyer of the products in this zone.
Why is copying needed? Consider an online furniture store based in Mysuru. One sofa order is recorded in four places. The store's app saves the order in its database. The payment system records the payment. The delivery partner's system tracks the delivery. If the customer returns the sofa, the return is logged in the support tool. On Monday, the head of sales asks how many sofas the store sold in Bengaluru last month, and how many came back. The answer needs data from all four systems, and each one stores its data in its own format.
So the data is copied into one place built for questions, the data warehouse or lakehouse.
The oldest way to copy it is ETL and ELT. ETL means extract, transform, load: pull the data out, clean it, then put it in the warehouse. ELT loads the raw data first and cleans it inside the warehouse. These tools rely on data connectors, ready-made code for pulling data out of each particular app. The cleaning step is called data transformation.
Copying on a schedule, for example every night, is called batch processing. The copy can then be a night behind. Change data capture (CDC) instead reads each change out of a database as it is written, so the warehouse is only seconds behind. Event streaming carries events, records that something happened, between systems as they happen. The page on batch and stream processing compares the two ways.
Embedded iPaaS sells connections to a different buyer: software companies, which build them into their own products for their customers.
Copying data for analysis is not the same as connecting applications. Connecting applications means that an action in one app starts an action in another, for example when a new hire in the HR system gets an email account. That is enterprise integration.
There are two ways to buy these tools. One suite can do everything, or the data team can join many single-purpose tools, called the modern data stack. Lately the separate tools have been merging back together. AI tools now also write pipelines and transformations from a written description, an idea explained on the page on AI coding agents. When a pipeline delivers late or incomplete data, data observability notices it.