High concurrency

High concurrency means that a very large number of people use a system at the same moment. "Concurrent" means happening at the same time. The hardest case is a sudden spike: millions of people arriving within the same minute. This happens in the last over of a cricket final, when a big concert's tickets go on sale, or when Tatkal train bookings open at 10 a.m. A system built for an ordinary day can freeze or crash at such a moment, unless it has been designed for it.

Why is a spike so hard? Each request needs a little of a server's time and memory, and one server can handle only so many requests at once. When too many arrive, requests wait in line, and the wait grows. Users who see nothing press Refresh, which sends even more requests. Soon the system may stop answering anyone at all.

Here is how engineers prepare for it. They spread the requests across many servers, using a load balancer. They keep ready-made copies of what most users ask for, so that it is not built again for each user. This is called caching. They send video and images from a content delivery network (CDN), a set of servers spread across the country, near the users. They add servers before the event. The cloud can add servers automatically, but each takes minutes to start, while a spike arrives in seconds. When even this is not enough, they put users in a queue and let them in at a rate the system can handle. They also switch off features that are not essential, such as comments, so that the main job keeps working. Before the event, they test the system with fake traffic far larger than they expect. This is called load testing.

Take, for example, a cricket final streamed online. In live streaming at scale, tens of millions of viewers join in the last few overs, and each of them must keep receiving the video without a pause.

Consider a concert ticket sale. A virtual waiting room gives each buyer a place in a line, lets a few thousand buyers in at a time, and turns away bots.

Say a state board publishes its exam results at noon. Lakhs of students open the results website in the same minute, and a site that has not prepared shows them an error page.

How is high concurrency different from a DDoS attack? A DDoS attack, short for distributed denial of service, also sends a flood of requests. But the requests come from computers controlled by an attacker, who wants to knock the system offline. The flood can look the same. The response is different: real users must be served, while attack traffic must be found and blocked. Blocking it is the job of DDoS protection. High concurrency is also different from high throughput, which is the total amount of work a system does over a day. A system can handle a crore requests spread across a day, and still fail when ten lakh arrive in one minute.