×

How AI Tech Is Changing Dubbing, Making Stars Like David Beckham Multilingual

David Beckham does not speak Arabic, Hindi or Mandarin. But when the soccer legend starred in a PSA for malaria awareness this spring, he effortlessly switched among these and six other languages, thanks to cutting-edge technology that could soon change how Hollywood localizes its movies and TV shows.

The PSA in question was produced with technology from Synthesia, a London-based startup that uses artificial intelligence for dubbing. In Beckham’s case, the company recorded video footage of the soccer star as well as native speakers in each of the languages it wanted to use.

Then it fed all that raw footage to an algorithm that “learned” the facial expressions for each word in languages like Spanish and Yoruba, and tweaked video of Beckham’s face accordingly. “You puppeteer video,” explains Synthesia co-founder and chief operating officer Steffen Tjerrild. “We have actual lip-sync.”

AI-based video editing has gotten a lot of attention for its darker side, best known as the “deep fake” phenomenon — porn videos that have been altered by so-called deep learning-based algorithms to convincingly feature the faces of celebrities. However, Synthesia’s work shows that the technology also has a lot of potential for Hollywood, with dubbing being a key area of interest.

Traditionally, dubbing has been done without altering the source video. Instead, local script writers aim to match translated dialogue to the action on screen. “It takes a lot to prepare those scripts,” says Tjerrild. What’s more, voice actors have to time their delivery perfectly to make sure the dub doesn’t feel off, a process that can take weeks for feature films.

Artificial intelligence that tweaks an actor’s mouth movements to fit the local language could significantly shorten those timelines and make dubs even more accurate, agrees Markus Gross, vice president of research at Disney Research. “If we had the possibility to change lip movement in post-production, that would be a huge deal,” he says. “It points to ways to make the lip-sync indistinguishable from what the actor would do if he spoke that language.”

Disney Research has done some work on using deep learning for speech animation, and is looking at ways to apply the tech to special effects. While dubbing isn’t a high priority for the studio’s research arm, Gross signals that he is intrigued by the possibilities. “It is definitely on the horizon for us,” he says.

Dubbing is becoming ever more important for Hollywood as media companies target consumers worldwide with their own streaming services. Netflix, for instance, is dubbing in 31 languages, and began to target English-language audiences with dubbed versions of foreign originals last year. “We are taking this new initiative very seriously,” says the company’s head of international dubbing, Debra Chinn, about its efforts to unlock global stories for the English-speaking world. 

The streaming giant has been exploring methods for automating some aspects of dubbing, but Chinn cautions that there may be limits to what technology can do. “Dubbing is an art,” she says. “It’s a creative process.”

There are indeed technical issues. Synthesia’s tech works best when actors look directly into the camera and is less effective with profile shots or action scenes.

Teaching the algorithms the facial idiosyncrasies of each individual actor is also challenging. “No actor can speak 20 languages,” says Gross. And while an algorithm can do a good job of making it look like someone’s mouth forms foreign-language words, it doesn’t know how an actor such as Leonardo DiCaprio would deliver a line in Mandarin.

Ultimately, studios might have to test their dubs with actual users, suggests Gross. “We have to expose the results to a lot of people and then ask them if they feel it’s authentic or if it looks weird.”

Employing AI for dubbing will be less demanding in animation, where the technology can also be used to better match the mouth movements of a character to the original language. “It is one important step to produce high-quality animation at a lower price tag,” says Gross. 

Those savings could help convince Hollywood to embrace this kind of technology in the near future. “Within the next two, three years,” Gross says, “we could see first applications.”

Popular on Variety

More Biz

  • Barron HiltonBarron Hilton 1990

    Famed Hotelier Barron Hilton Dies at 91

    Barron Hilton, a famed hotelier who helped expand the Hilton Hotels empire and a founding owner of the Chargers NFL football team, has died, the Conrad N. Hilton Foundation announced. He was 91. “Today the world of hospitality mourns for one of the greats. Barron Hilton was an incredible family man, business leader and philanthropist. [...]

  • Patrick Whitesell and Ari Emanuel WME

    Endeavor Targets Sept. 27 for Stock Debut, IPO Video Tells Company's Origin Story

    After years of preparation, Endeavor is set to make its formal Wall Street debut on Sept. 27, when its stock will begin trading on the New York Stock Exchange. Endeavor has targeted Sept. 26 for the final pricing of its shares. The stock will trade publicly the following day. Earlier this week, Endeavor said its [...]

  • Netflix - Apple TV

    Netflix Stock Drops After CEO Acknowledges 'Tough Competition' Coming From Disney, Apple

    Netflix shares fell as much as 7% Friday to a nine-month low, coming after CEO Reed Hastings commented that the November launches of Disney Plus and Apple TV Plus will introduce a “whole new world” of competition. Hastings, speaking at the Royal Television Society conference Friday in Cambridge, England, said, “While we’ve been competing with [...]

  • Charlie Rose Sexual Harassment

    Charlie Rose Sued for Sexual Harassment by Longtime Makeup Artist

    A makeup artist who worked for Charlie Rose for 22 years has filed a sexual harassment lawsuit, accusing the former CBS and PBS host of years of unlawful behavior toward female employees. Gina Riggi alleges that Rose was verbally abusive with her and would often make derogatory comments about her weight. She also alleges that [...]

  • Rob Stringer

    Sony Music Chief Rob Stringer on Sustaining Growth and Recovering From the 'Dark Times'

    The Goldman Sachs Communacopia conference, now in its 28th year, gives top executives at major companies the opportunity to make their case to investors — and the Goldman analysts the opportunity to keep things on the up and up. While the analysts don’t necessarily grill the executives, they don’t lob softball questions either. That was [...]

  • Frank Grillo'Avengers: Endgame' Film Premiere, Arrivals,

    Matt Phelps Tapped as President of Joe Carnahan, Frank Grillo's Warparty

    Frank Grillo and Joe Carnahan’s Warparty productikon banner has appointed Matt Phelps president of the company. Phelps will head the Los Angeles office and be responsible for overseeing all film and television projects. “We searched long and hard to find the right fit for Warparty and felt that Matt embodied everything that we were looking [...]

  • Jack Gilardi, Longtime ICM Partners Agent,

    Jack Gilardi, Longtime ICM Partners Agent, Dies at 88

    Jack Gilardi, a longtime ICM Partners agent who represented such stars as Burt Reynolds, Sylvester Stallone, Jerry Lewis, Charlton Heston and Shirley MacLaine, died Thursday at his home in Los Angeles. He was 88. Gilardi was known for his gentlemanly style, love of the Los Angeles Dodgers and his skill at representing top actors. He [...]

More From Our Brands

Access exclusive content