Tech, ExplainedData platforms and AI
Product

Key-value store

One key, one answer, millions of times a second.

BEFORE ITAFTER ITKey-value storeAerospikeCOVERSLookup by keyUnder 1 msIn memoryReplicatedEvent streamingupdateseach profilereturns aviewer'sprofileAd exchange↗ PROGRAMMATIC ADVERTISINGCONCEPTS IT USESCachingBEFORE ITAFTER ITKey-value storeAerospikeCOVERSLookup by keyUnder 1 msIn memoryReplicatedEvent streamingupdateseach profilereturns aviewer'sprofileAd exchange↗ PROGRAMMATICADVERTISINGCONCEPTS IT USESCachingBEFORE ITEvent streaming↓ updates each profileKey-value storeAerospikeLookup by keyUnder 1 msIn memoryReplicatedAFTER IT↓ returns a viewer's profileAd exchange↗ PROGRAMMATIC ADVERTISINGCONCEPTS IT USESCaching
The key-value store receives profile updates from event streaming and returns a viewer's profile, found by its key, to the ad exchange in under a millisecond.
opens a page inside this one a page in the same scene a page in another scene a concept page a label only, with no page

A key-value store is a database that saves each record under one unique key, and finds it by that key alone. It is the simplest kind of database, and the fastest. A well-built key-value store answers in under a millisecond, millions of times a second.

The key is a name or number, such as a customer ID. The value is whatever is stored under it: a number, a short text or a small document. To read a value, the program must know the key. It cannot ask the store to search inside the values, for example for every customer in Pune. That limit is what makes the store fast. The store always knows exactly where each key's value is kept, so it goes straight to it.

Most key-value stores keep their data, or at least the list of where each key is, in memory, which is far faster than reading from disk. The page on caching explains why. The data is also copied to several machines, so that the store keeps answering if one machine fails.

Imagine a reader in Nagpur who opens a cricket score page on her phone. Before the page finishes loading, an ad exchange auctions the ad space beside the score. The whole auction must finish in about 100 milliseconds. The bidding software looks up the phone's advertising ID, a random identifier for the device, in a key-value store. The key is the advertising ID. The value is a short profile: interested in cricket, two visits to car websites this week. The lookup takes less than one millisecond, which leaves time to decide on a bid. During a big match, the same store may answer millions of such lookups every second. The page on programmatic advertising explains the auction itself.

Payment systems use key-value stores in the same way. A payment fraud check may read, by card number, how many payments that card made in the last hour, within the few milliseconds it has.

The profiles have to be kept up to date. New clicks and visits arrive as events through event streaming, and each event updates the value stored under its key.

How is a key-value store different from a cache? A cache holds a temporary copy of data that lives somewhere else, and it may lose that copy. A key-value store can be the main place where the data is kept, and it must not lose it. How is it different from a document database? A document database can also search on the fields inside each record. A key-value store gives up that search in return for speed.

Tech, Explained is a part of ingrid.fyi. Every page explains one piece of technology in plain words.