Log management is the system to which all of a company's servers send their logs, so that engineers can keep them in one place and search them in seconds. It is also called log search, log analytics or centralised logging. A log is the text that a program writes as it runs, one line for each event, such as "14:02:31 ERROR payment failed for order 1234: card declined". Every program writes logs: the app, the database, the web server, the operating system and the firewall.
Why collect them in one place? Without log management, each server keeps its own log files on its own disk. To investigate a problem, an engineer has to log in to each server and read its files one by one. A company with 300 servers cannot do that during an outage. Also, servers are often replaced, and their log files disappear with them.
Log management works in four steps. On each server, a small program called a shipper reads each new log line as it is written, and sends it over the network. The central system breaks each line into fields, such as the time, the server's name, the error level and the order number. This step is called parsing. It then stores the lines in an index. An index works like the index at the back of a book: a search can jump straight to the matching lines instead of reading every line. Finally, engineers search, count and draw graphs of the lines.
Consider an IT services company in Pune that runs the loan app of a client, a lender licensed as an NBFC. At 1:40 a.m., the client reports that loan applications are failing. The app runs on 300 servers, which together write about 2 crore log lines an hour. The engineer types one search: the word "ERROR", in the loan service, in the last hour. In three seconds, the system finds 1,840 matching lines. She groups them by their message. Almost all of them say "timeout calling credit bureau". She counts them by minute. They began at exactly 1:10 a.m. So the fault is not in the client's code. The credit bureau's service, which reports each applicant's credit score, has become slow. She informs the client within 20 minutes, with the counts as evidence.
How long are logs kept? Usually for a fixed time, such as 30 days in fast storage for searching, and a year in cheaper storage. Some of these rules come from the government. For instance, CERT-In, India's national agency for cyber security incidents, requires companies to keep the logs of their systems for 180 days.
Log management is a part of the observability platform, next to metrics and traces. Logs are one of the three kinds of telemetry. The same log lines are valuable to the security team too. So they are often also sent to a SIEM, which searches them for signs of an attack instead of signs of a fault. Searching logs is usually the first thing an engineer does when a live system breaks.