Tech, ExplainedData platforms and AI
Product

Real-time analytics database

Billions of rows answered in under a second.

BEFORE ITAFTER ITReal-timeanalytics databaseClickHouseImplyApache DruidCOVERSSub-second queriesStreaming ingestColumn storageMany usersEvent streamingsends eachnew eventsupplieslive figuresBusiness intelligence(BI) dashboards↗ BUSINESS INTELLIGENCECONCEPTS IT USESColumnar databaseBatch vs stream processingOpen source business modelBEFORE ITAFTER ITReal-timeanalytics databaseClickHouseImplyApache DruidCOVERSSub-secondqueriesStreamingingestColumn storageMany usersEvent streamingsends eachnew eventsupplieslive figuresBusinessintelligence(BI) dashboards↗ BUSINESSINTELLIGENCECONCEPTS IT USESColumnar databaseBatch vs stream processingOpen source business modelBEFORE ITEvent streaming↓ sends each new eventReal-time analytics databaseClickHouseImplyApache DruidSub-second queriesStreaming ingestColumn storageMany usersAFTER IT↓ supplies live figuresBusiness intelligence (BI) dashboards↗ BUSINESS INTELLIGENCECONCEPTS IT USESColumnar databaseBatch vs stream processingOpen source business model
The real-time analytics database receives each new payment from event streaming within seconds, and answers live dashboard queries over billions of rows in under a second.
opens a page inside this one a page in the same scene a page in another scene a concept page a label only, with no page

A real-time analytics database is a database that answers questions over billions of rows in under a second, including over data that is still arriving, such as clicks on a website or payments every second. It is also called a real-time OLAP database. Like a data warehouse, it stores data column by column. But it is built for speed, and for thousands of people asking at once, rather than for long reports.

Why is a warehouse not enough? A warehouse is usually loaded in batches, and a large query can take several seconds or minutes. That is fine for a monthly sales report. It is too slow for a screen that thousands of users refresh every few seconds.

How does it answer so fast? It stores each column separately and compressed, as a columnar database does. So a question about payment amounts reads only the amount column. It splits the data across many servers, and each server scans its own part at the same time. It often keeps running totals, such as the number of payments in each minute, and updates them as each row arrives. Then a total does not have to be added up again for every question. It also reads new rows directly from an event streaming system, within a second or two of each event. The difference between loading data in batches and reading it as it arrives is explained in batch vs stream processing.

Say a payment company in Noida gives each of its 5 lakh shop owners a sales dashboard inside its app. At 8 p.m. on Dhanteras, the owner of a utensils shop in Patna opens it. She sees today's sales, her failed payments and her busiest hour, correct to the last minute. That one screen runs six queries on a table of 400 crore payments, filtered to her shop. The database must answer each query in about 200 milliseconds, while 20,000 other shop owners open the same screen. A warehouse would be too slow, and too costly, for this many small queries.

Companies also use these databases for their own live operations screens, and for BI dashboards that need figures from the last few minutes.

Most products of this kind are open source. The companies behind them earn money by running the database as a cloud service and by selling support. This pattern is explained in open source business model.

How is it different from an HTAP database? An HTAP database records the original transactions itself, and also answers analytical questions on the same data. A real-time analytics database does not record the original transaction. It receives a copy of each event, and only answers questions about them.

Tech, Explained is a part of ingrid.fyi. Every page explains one piece of technology in plain words.