You order a pair of running shoes on a shopping app in Chennai. The moment you tap Pay, the app's backend, the part that runs on the company's servers, writes the order into a database. In the same instant, the stock count for those shoes drops by one. Later, a copy of your order lands in a second, much bigger store. There, the company adds up millions of orders to see what sold, where and when.
These are the two lives of every piece of data, and this Market Segment follows both. The first is recording each event as it happens: an order, a message, a bank balance. The industry calls this transactional work, or OLTP, short for online transaction processing. The second is asking questions across all the records at once. This is analytical work, or OLAP, short for online analytical processing.
Every piece of data has two lives: the app's database records the shoe order the moment you tap Pay, which is transactional work, and a copy goes on to a warehouse that adds it up with millions of others, which is analytical work.
Databases are one of the oldest kinds of software product, and one of the most fought over. The reason is lock-in. A company's code and years of its data are built around one database. Moving to another means rewriting code and copying all that data without losing any. The industry calls these switching costs. So the choice of database is the one a company finds hardest to reverse.
Who buys, and how the money flows. The buyers are the companies that build software, from banks and shopping apps to start-ups. Many of the databases here are open source: anyone can download the code and run it for free. So the companies behind them earn in three main ways:
A paid versionThe free version is the core, and the features large companies need are sold on top. The industry calls this open core.
Running it for youThe seller runs the database in the cloud, and the buyer pays for what it uses. This is called a managed service, or database as a service.
SupportThe buyer pays for expert help when the database breaks or slows down.
Almost every company here is headquartered abroad and does engineering work in India. Three of the largest databases in the world are missing. Oracle Database is not covered in this Market Segment. Microsoft's SQL Server and Google's BigQuery are covered elsewhere. (More on them in Microsoft and Google, in the big-tech collection.) The free databases that many new applications use, such as PostgreSQL, appear only through the companies that sell around them.
Every company named here has posted software engineering jobs in India. Famous companies that don't actively hire software engineers in India are left out.
This Market Segment has three sub-segments. The first covers the databases that apps write to. The second covers the warehouses and lakehouses, where the copies go to be analysed. The third, a short one, covers two companies that make no database of their own but run other people's. The disks and backups underneath every database are covered in Market Segment 7.2, Enterprise storage and backup.
The database the application writes to
The databases an app reads and writes as it runs: document, key-value, distributed SQL, graph and relational
Take the shoe order from the start of this Market Segment. It was written to this kind of database, the one the app talks to every second.
You probably know the relational database from college: tables of rows and columns, read with SQL. MySQL is one, and so are Oracle's and Microsoft's. For decades, almost every application ran on a single relational database.
Since around 2010, web apps have grown to millions of users. One database server could no longer keep up, and fixed tables were slow to change as the app changed. So a new group of databases appeared that don't use tables. The industry calls them NoSQL databases.
MongoDBsells a document database. It stores each record as a document, much like the JSON an app already sends and receives. A programmer can save the shoe order as one document, with the shoes, the address and the payment inside it. MongoDB became one of the most widely used databases for web applications. It has engineers in Delhi NCR and Bengaluru.
Couchbasealso sells a document database, with a cache built in. A cache is a copy of the most-used data, kept in memory so it can be read very fast. Couchbase is for applications that need an answer within a millisecond. It has engineers in Bengaluru.
Aerospikealso in Bengaluru, goes further in the same direction. It is a key-value store: each record is saved under one key, and found by that key alone. It is built for advertising exchanges and payment systems, which must answer millions of times a second. An advertising exchange is the system that auctions the ad space on a web page while the page loads.
Newer companies have brought the relational database back, rebuilt to run across many machines. The industry calls this distributed SQL.
Yugabytewas founded by engineers who built the databases inside Facebook. From Bengaluru, it sells a database that works like PostgreSQL, the popular free relational database. An app written for PostgreSQL can use it without changes. But Yugabyte spreads the data across many machines, and even across cloud regions, which are data centres in different cities or countries. The application doesn't notice.
SingleStorewith engineers in Pune, joins the two lives of data from the start of this Market Segment. It handles transactions and analytics in a single engine. So a company can ask questions of live data, without waiting for a copy to reach a warehouse.
The classic relational database is still a business too. MariaDB is the company behind MariaDB, a fork of MySQL. A fork is a copy of a free program's code that a separate team develops on its own. MySQL is owned by Oracle. Some of MySQL's original developers started MariaDB in 2009, worried about MySQL's future once Oracle took it over. A separate non-profit foundation looks after MariaDB's open-source code.
Two more companies sell different kinds of database altogether:
Neo4jsells a graph database. It stores records and the links between them, and the links are the point. That suits questions like "which of these accounts are connected?" It is used to find fraud rings, groups of accounts working together to cheat. It is also used for recommendations. More recently, it is used to give AI chatbots facts to base their answers on. (More on this in Market Segment 9.2, MLOps and LLMOps platforms, in Industry Vertical 9, AI labs and model platforms.)
MonkDBa Hyderabad start-up founded in 2023, presents itself as one database for many kinds of data. It covers documents and graphs. It also covers time series, which are readings stamped with the time, such as sensor data. And it covers vectors, the lists of numbers that AI models use to capture the meaning of text or images.
The same order can be stored as rows in relational tables, as one document holding everything, as a value found by a single key, or as points in a graph where the links between records are the point.
Several other database companies are not covered in this Market Segment: Redis, EDB, Cockroach Labs, DataStax, and the new PostgreSQL cloud services Supabase and Neon.
The warehouse and the lakehouse
Data warehouses, big-data platforms, lakehouses and fast analytical databases
Back to the shoe order from the start of this Market Segment. A copy of it, along with every other order, is sent on to a data warehouse. (More on how the data is copied in Market Segment 8.1, Data integration and governance platforms, in Industry Vertical 8, Analytics and data providers.) In the warehouse, the company can ask questions across all its orders, such as which shoes sold best in Chennai last month.
Teradatais one of the companies that built the enterprise data warehouse. This is the system into which a bank or a retailer copies every transaction, so that analysts can ask what happened across all of them. Its Indian engineers work on its Vantage platform from Hyderabad, Bengaluru and Pune.
Clouderagrew from Hadoop, the open-source system for storing and processing data too large for one machine. It now sells a data platform that runs both in the cloud and on-premises, meaning on the company's own servers. It has engineers in Bengaluru and Kolkata.
The first warehouses ran on the company's own servers. Then came the cloud warehouse, rented as a service, with the company paying for the computing its questions use.
Snowflakeis one of the companies that defined the cloud warehouse. It builds part of its data platform in Bengaluru. (More on the charts and dashboards built on top of warehouses in Market Segment 8.2, Business intelligence (BI) and analytics platforms, in Industry Vertical 8, Analytics and data providers.)
A warehouse is not the only place data is kept for analysis. A data lake is cheap storage where a company keeps raw files of every kind, as they are. A warehouse holds clean, structured tables, and costs more.
Databrickshas engineers in Bengaluru. It began calling the combination of the two a lakehouse: one system with the low cost of a lake and the structure of a warehouse. It now also sells the tools to build AI models on the same data. (More on those tools in Market Segment 9.2, MLOps and LLMOps platforms, in Industry Vertical 9, AI labs and model platforms.)
A data lake keeps raw files cheaply and a warehouse keeps clean tables at higher cost; the lakehouse combines the two, and open table formats keep a company free to move its data to another tool.
Two more companies sell speed: answers in under a second.
ClickHouseis an open-source column store, sold as a cloud service. A column store saves data column by column, not row by row. To add up one column, such as the price of every order, it reads only that column. So it answers analytical questions over billions of rows in under a second. It has engineers in Bengaluru.
Implywas founded in 2015 by the creators of Apache Druid, and it sells Druid. Druid is an open-source database built for the same fast questions, over streaming data. Streaming data arrives nonstop, like the clicks on a website.
To add up the price of every order, a row store reads every row in full, while a column store reads only the price column, which is why it can answer over billions of rows in under a second.
The last company here goes back to the lakehouse. Onehouse was founded by the creator of Apache Hudi, and has engineers in Bengaluru. Hudi was first built at Uber. It is one of the open table formats: free, public rules for storing a warehouse's tables as ordinary files that any tool can read. Onehouse sells a managed lakehouse in these open formats. Because the tables are in an open format, a company is not tied to any one warehouse.
Firebolt, Dremio, Starburst and Exasol are not covered in this Market Segment.
The lakehouse as the AI platform.
Warehouses were built to feed reports and dashboards. Lakehouses are now also used to train and run AI models. Databricks, Cloudera and Teradata each make the same pitch: the place where a company's data already lives is the place to train and run its models. That brings back the lock-in question from the start of this Market Segment. The open table formats that Onehouse works with are what let a company move its data later if it wants to.
You're reading as a guest. Sign in free to follow links for five minutes, once an hour.
Sign in
Running someone else's database
Open-source database support, database as a service
Back to the shopping app from the start of this Market Segment. The company behind it may run a free database, like MySQL or PostgreSQL, itself. Then it needs someone to call when the database slows down during a big sale. Or it may not want to run the database at all. Two companies here make no database of their own. They help run other people's.
Perconasells support and tools for the open-source databases that a company runs itself: MySQL, PostgreSQL and MongoDB. Support means expert help when a database breaks or slows down. Percona has engineers in Bengaluru.
Tessellsells database as a service across the three big clouds: Amazon Web Services, Microsoft Azure and Google Cloud. It runs Oracle, PostgreSQL, MySQL and SQL Server for its customers. It handles the backups, the patching and the performance. Patching means installing updates and security fixes. The cloud providers do this job for their own databases. Tessell does it for any of them. (More on the cloud providers' own database services in Market Segment 5.1, Cloud infrastructure and web hosting providers, in Industry Vertical 5, Cloud providers and OS makers.) Tessell has engineers in Bengaluru and an office in California.
That is the life of the shoe order. The app's database records it the moment you tap Pay, and the warehouse counts it with millions of others. Sometimes, a company that did not make the database is the one keeping it running.