Mann was inspired to create Flawless after he saw a foreign-language dub of “Heist.” The director remembered how obsessive De Niro was in the creation of the character, going so far as to analyze what cufflinks would be best for his character to wear. But all the specific choices De Niro made to create the character went out the window with a bad dub.
“The dubs effectively removed the way [De Niro] made character decisions, by changing dialogue to fit his facial movements, instead of adapting the script and staying true to his performance,” Mann said. “It became apparent that the process doesn’t work. At best, it’s a compromise.” Per Input: “[Flawless’ technology] strips actors’ faces off, converting their visages into a 3D model, according to Lynes. ‘This creates millions of 3D models, which the AI uses as reference points,’ he says. ‘And then, using an existing foreign-language recording of the dialogue, it studies the actor and generates a new 3D model per frame.’ Finally, the imagery is converted back to 2D. Digital effects artists can then manually fix anything that seems off.” The Flawless method moves the dubbing process out of post-production. Dubbing audio is recorded at any point in production and is used in tandem with the 3D models created by the AI to synch up the dub with the proper mouth movements. While the system recalls deepfake technology, Mann stressed it is a much different process. “Deepfakes use two-dimensional face-replacement to hijack people’s identity,” Mann said, “which is the opposite of what we do, which is responsibly utilizing neural networks to enhance a performance. We also take into account the geometry and geography of the image, which is the biggest difference — this creates a fully three-dimensional performance.” Mann told Input that Flawless is in talks with film studios and is currently working with one streamer. The first movie to be released using the Flawless technology should arrive sometime in the next 12 months. “There will be people who will refuse to watch our dubs and say subtitles are the purest form, but this helps share great international stories with those who may have never otherwise seen them,” Mann added about Flawless. “We’ll soon reach the level where you won’t realize we’ve done it, and the debate will solve itself.”
Flawless is far from the only company looking to help Hollywood solve its subtitles problem. IndieWire reported in January on Deepdub, an Israel-based startup that claims that it can accurately dub an actor’s voice into other languages using just a few voice samples while retaining the actor’s tone and pitch. Head over to Input Magazine’s website to read more on Flawless. Sign Up: Stay on top of the latest breaking film and TV news! Sign up for our Email Newsletters here.
