Computer vision

Computer vision is software that recognises objects in photos and video. It is used to check shelves and products in shops, to find defects in a factory, to read number plates and to see what a robot or a shopper picks up. Its output is usually a list of what was found in the picture and where.

Here is how it works in general. A camera takes a picture, which is a grid of numbers for the colour of each dot. A model trained on many thousands of labelled pictures looks at these numbers. During training, people marked the objects in each picture, so the model learned which patterns of dots mean a bottle, a crack or a face. In use, the model finds the same patterns in a new picture and says, for example, "a bottle at this position, 92 per cent sure". Software then acts on that result.

There are four common tasks. Classification says what is in a picture. Detection also says where, by drawing a box. Segmentation marks the exact outline of each object. Text reading, called OCR, turns printed letters into text. The model is only as good as its training pictures. If it has never seen a type of damage or a type of lighting, it may miss it. It gives a confidence score, and the user picks the limit below which a person must check. Labelling the training pictures costs time and money.

Take, for example, a camera above a shop shelf. It counts the packets of biscuits and alerts staff when a gap appears; see shelf gap detection and shelf image recognition. Consider a factory line, where a camera checks that every bottle has a cap and a label; see machine vision. Imagine a store with no cashier. Cameras see which items a shopper picks up and add them to the bill; see checkout-free stores.

Other uses are ANPR, video analytics, drone vision and radiology AI. It runs either in the cloud or near the camera, as in edge computing. The models are often built on a foundation model.

How is it different from speech recognition? Computer vision works on pictures and video. Speech recognition works on sound.