Automated testing

Automated testing is the practice of checking software with small programs, called automated tests, instead of only by hand. Each test uses the app the way a person would, such as typing a phone number or pressing Pay, and checks that the result is the one expected. The tests run again automatically after every change to the code. So if a change breaks something that worked before, the team finds out within minutes, and not from its customers. A break of this kind is called a regression.

Here is how it works in general. A developer writes a test once. The test sets up a situation, performs an action and compares what happened with what should have happened. For example: add two items worth ₹200 and ₹350 to the cart, and the total shown must be ₹550. If the total is anything else, the test fails. Tests come in several sizes. A unit test checks one small piece of code, such as the function that adds up a bill, in a fraction of a second. An API test sends a request to the app's server and checks the reply. An end-to-end test opens the app in a real browser or phone and clicks through a whole task, such as booking a ticket. End-to-end tests catch more problems, but they are slow and break more often, so teams write many unit tests and fewer end-to-end tests.

When a developer proposes a change as a pull request, a CI/CD pipeline runs all the tests on it automatically. If any test fails, the change is stopped, and the developer sees which test failed and why. Only a change that passes moves on towards production, the live system that customers use.

Automated tests have costs too. They must be updated whenever the app is meant to change. A flaky test, one that sometimes passes and sometimes fails with no change to the code, wastes hours of engineers' time. In India, testing became the largest single job in the software industry long before testing tools became products.

Take, for example, a banking app that must work on many models of phone. A real device testing service runs the same tests on hundreds of real phones at once.

Consider a company that has customised its ERP software, the system that runs its accounts, purchases and payroll. ERP testing reruns the company's tests after each quarterly update from the vendor, to find what the update broke.

Say an AI coding agent writes a change. It runs the tests itself, reads any failure, and corrects its code before a person reviews it.

How is automated testing different from testing non-deterministic software? An ordinary automated test expects one exact answer, such as ₹550. An AI chatbot gives a slightly different answer every time. So its tests must judge whether each answer is acceptable, instead of matching it word for word.