Anomaly detection

Anomaly detection means learning what a stream of readings normally looks like and flagging anything that departs from it. The readings can be a server's load, a turbine's temperature, a bank account's transactions or a factory machine's sound. An anomaly is a reading, or a pattern of readings, that does not fit what has been seen before.

Here is how it works in general. First, the system collects a long run of readings from the time when everything was working well, usually as time series data. It builds a picture of normal, such as the usual range at each hour of the day. As new readings arrive, the system compares each with that picture and gives it a score for how unusual it is. If the score is above a limit, the system raises an alert. A person then checks whether it is a real problem or a harmless change. The checks teach the system, and the limit is adjusted so that there are fewer false alarms.

There are three common methods. The first is statistical: it finds the usual average and spread, and flags anything far outside it. The second makes a forecast of the expected value and compares the reading with it. The third learns from examples of normal readings. There is a trade-off in the limit. If it is set low, the system raises many false alarms. If it is set high, it misses real problems. Normal also changes over time, with seasons, new settings and ageing, so the picture of normal must be refreshed.

Take, for example, a wind turbine whose gearbox temperature normally rises and falls with the wind. If the temperature rises while the wind is calm, the system flags it; see wind turbines. Consider a cloud company that watches the response time of its website and is alerted when it doubles; see observability platforms. Say a bank sees a card used in two cities within an hour; see transaction monitoring.

The method is the first step of predictive maintenance, and it supports security monitoring through user behaviour analytics. It also helps in data observability.

How is it different from a fixed alarm? A fixed alarm triggers when a reading crosses one number, such as 90 degrees. Anomaly detection learns what is normal for that machine at that time, so it can catch a problem before any fixed number is crossed.