Observability

Observability is the ability to find out what a running software system is doing, and why, from the records that the system keeps about itself. These records are called telemetry. They come in three kinds: metrics, logs and traces. The word observability is also used for the industry that sells the tools to collect and search these records.

Here is how it works in general. A modern app runs on many servers. A server is a computer in a data centre that runs programs for other computers. Nobody can see what these programs are doing just by looking at the servers. So developers instrument the code: they add a few lines that record what happens. Metrics are numbers measured over time, such as the number of requests each minute, or how many milliseconds each one takes. Logs are lines of text that a program writes as it runs, such as "payment failed for order 50417: bank timed out". Traces follow one request, such as one customer's click, through every service it passes on the way. A service is one small program that does one job, such as checking stock or taking payment. All three kinds of telemetry are sent to an observability platform. It stores them, draws graphs and raises an alert when a number crosses a limit. An alert is a message, usually to an engineer's phone, saying that something needs attention. The engineer then searches the logs and traces to find the cause.

Take, for example, a train-ticket booking website at 10 a.m., when advance bookings open. Its metrics show requests rising from 2,000 to 40,000 a minute, and the response time rising with them. This tells engineers to add servers before customers start to give up.

Consider a hospital's lab-report app that fails for some patients. A search in log management shows the same error on every failed request: the program cannot read reports larger than 10 MB.

Say a mobile network operator wants to know why calls drop in one district. Its radio equipment sends telemetry too, and anomaly detection picks out the one tower whose readings changed after an update.

How is observability different from monitoring? Monitoring is the older idea. It checks a fixed list of things that engineers decided on in advance, such as "is the server running?" or "is the disk 90% full?" It answers known questions. Observability keeps enough detail to answer a new question that nobody thought of in advance, such as "why are payments slow only for one bank, only on Android?" Monitoring still exists inside observability, in tools such as infrastructure monitoring. And when the alert comes at night, an AI SRE can now read the telemetry first. All of this happens in production, the live system that customers use.