AI dubbing and lip sync is software that changes an actor's lip movements on screen to match the words of a dubbed language. So a film dubbed into Tamil looks as though it was shot in Tamil. It is also called visual dubbing. Some tools also create the dubbed voice itself, in a voice that sounds like the original actor's.
Dubbing means replacing a film's dialogue with the same lines in another language. In ordinary dubbing, voice artists record the new lines in a studio, while they watch the film. They try to match the length of each line. But the actor on screen is still saying the original words, so the lips do not match. Viewers notice this most in close-ups.
Here is how lip sync works. The software first finds the actor's face in every frame, and tracks the mouth, jaw and cheeks. Next, it listens to the dubbed audio and works out which mouth shape each sound needs. For example, the sounds "p" and "m" need closed lips, and "o" needs round lips. An AI model then generates a new lower face for each frame, with the mouth shapes of the new language. It blends this into the original picture, keeping the actor's skin, the lighting and any beard or make-up. Finally, an editor checks the result and fixes any frames that look wrong.
Take, for example, a Malayalam family drama that its producer wants to release in Hindi as well. The film is 2 hours and 20 minutes long and has about 900 shots with dialogue. Hindi voice artists record the dub as usual. The lip-sync software processes the shots in which a face is clearly visible, about 600 of them, on rented servers. Over three weeks, editors review every changed shot and send about 40 back to be redone. The Hindi version is then released on a streaming service, where it looks like a Hindi film.
Why does this matter in India? Many Indian films are released in four or five languages at once. Many viewers prefer to watch in their own language, as described in vernacular internet. A dub that looks natural can bring a film to many more viewers. The same tools are used for short episodes on a micro-drama app, and for advertisements made in several languages.
The technology also raises questions. Changing an actor's face, or copying an actor's voice with speech and voice AI, needs the actor's consent and a contract that says how it may be used. The same methods can be misused to make fake videos of real people, which is why deepfake detection software exists. These questions apply to all AI-generated content.