Event streaming is software that carries events between a company's systems as they happen. An event is a record that something happened, such as an order placed, a click or a payment. The most widely used software of this kind is Apache Kafka, which is free and open source. It began inside LinkedIn, and its creators later started a company that sells it as a cloud service.
The system that creates an event is called a producer. It writes the event once, to a named log called a topic, for example "orders". The streaming software stores the events of each topic in the order they arrived, and keeps them for a set period, such as seven days. Any number of other systems, called consumers, read the topic. Each consumer reads at its own pace and remembers how far it has read. A busy topic is split into parts called partitions, spread across many machines, so that millions of events a second can be handled.
This design means the producer does not need to know who uses its events. A new system can start reading tomorrow, without any change to the producer.
Consider a food delivery app in Kolkata at 8:30 p.m. A customer orders a ₹420 biryani. The order service writes one event to the orders topic. Within milliseconds, five consumers read it. The restaurant's tablet shows the order. The rider assignment service starts looking for a rider nearby. The notification service sends the customer a message. The fraud check looks at the payment. And an analytics loader adds the order to a real-time analytics database, which shows live orders per area per minute. Each rider's app also sends a location event every few seconds to another topic. That evening, the notification service fails for ten minutes. When it restarts, it carries on from the last event it read, so no customer misses a message.
Events also arrive from other sources. Change data capture turns every change in a database into an event. An industrial data platform sends readings from machines. Consumers can keep the latest value for each key in a key-value store, and keep every event, for analysis later, in a data lakehouse.
How is it different from a message queue? In a queue, each message is usually removed once one consumer has read it. In event streaming, the events stay for the set period, so many consumers can read them, and a consumer can read them again from the start. And how is it different from copying once a night? The page on batch and stream processing compares the two.
Companies can run Kafka free on their own servers, or pay for a cloud service and support. The open source business model explains how the company behind it earns.