Batch vs stream processing

Batch vs stream processing is the choice between two ways of moving and handling data. In batch processing, new records are collected and handled together, at fixed times, such as once every night. In stream processing, each record is handled within seconds of the moment it is created. So a copy of the data made by batch processing can be up to a night behind, while a copy made by stream processing is only seconds behind. Stream processing is also called real-time processing, or streaming.

Here is how batch processing works. A company's apps record events all day: orders, payments, logins and clicks. An event is a record that something happened. At a fixed time, say 2 a.m., a program called a batch job starts. It picks up every record that is new or changed since the last run. It copies those records, cleans them and loads them into the destination, such as a data warehouse. Then it stops until the next run. Most ETL and ELT pipelines work this way. Batch processing is simple and cheap. If a run fails, an engineer fixes the problem and runs it again.

Here is how stream processing works. Each event is sent onward the moment it is written. Change data capture reads each change out of an app's database as it is saved. Event streaming software carries these events, in order, to every system that needs them. The programs that receive the events run all the time, and act on each event within seconds. Streaming is harder to run than batch. The programs must never stop. Some events arrive late, or arrive twice, and the programs must still count each one exactly once. Streaming also costs more, because its computers run day and night.

Many companies use both. There is also a middle path, called micro-batching, in which a small batch runs every few minutes.

Take, for example, the interest on savings accounts. A bank's core banking system adds each day's interest to every account in one nightly batch, because nobody needs the figure before morning.

Consider a card payment that may be fraud. Payment fraud detection must score each payment within a fraction of a second, while the customer waits at the counter, so it works on a stream.

Say a quick-commerce app wants to show its operations team the orders placed in each area in the last five minutes. A real-time analytics database answers this from order events that arrive nonstop.

How is batch vs stream processing different from OLTP and OLAP? OLTP and OLAP describe what a database is doing: recording each event, or answering questions across all the events. Batch and stream describe when the data moves from one system to the next, and how often. A warehouse used for OLAP can be filled by either method. The choice depends on one question: how old can the answer be? A monthly sales report can wait a night. A fraud check cannot wait even a minute.