In 2015, director Martin Scorsese admitted that he was mulling over one story for a while. The trouble was, without the right kind of technology, it couldn’t be done the way he wanted to do it. That movie was The Irishman, starring Robert De Niro, Joe Pesci and Al Pacino.
The aforementioned problem was de-ageing the actors for their roles in the movie. Fortunately, Industrial Light & Magic (ILM) – the visual effects arm of Lucasfilm– came to the rescue.
VFX supervisor, Pablo Helman was the one to pitch the de-ageing idea, and Scorsese rolled with it. Helman got a copy of the script of The Irishman that night. By this point, Scorsese had been sitting on the film for about six years. Helman came back to set the next day saying he was in.
“It was the perfect opportunity for visual effects,” he says. “[I’m] always looking for a project that has visuals that support the content all the way.” In other words, there are plenty of VFX-laden stories out there created primarily for spectacle.
But for a veteran VFX artist, monsters and superheroes are just the day job. The dream for Helman was a project where the VFX supports the reality-based story. Proving he’s not as much of a magician as he seems, the two-time Oscar nominee (third time’s a charm) was good enough to reveal how his trick works
He and Scorsese knew that it was imperative to preserve the performances. Neither of the actors should wear markers on their faces nor a helmet with little cameras in front. This was primarily because they were method actors, so that was exactly the kind of actor-driven opportunity that [ILM] had been looking for, to bring in this innovative tech with an incredible project.
Back in 2015, with the huge CGI craze, everybody was wearing hundreds of markers on their faces, a helmet cam and grey pyjamas. They’d be in a room not even with their acting partner, just someone standing there for an eye-line.
They wanted to ignore all the markers, and just roll with the cameras and actors. There are three cameras on the rig designed with director of photography Rodrigo Prieto, ASC, AMC, and ARRI Los Angeles.
The centre camera is the director camera and to the left and right of it there are two infrared “witness” cameras. On the infrared cameras, there are two infrared rings that illuminate the actors.
Then, the software [called FLUX] takes a look at the images combined from the three cameras and it creates deformation, or geometry that mimics the actor’s performance. It takes a look at one frame and then it renders it, and then takes a look at the next frame and renders it and so on, until it completes the scene.
Because there are no markers to be tracked, the software is a lot more sensitive to the way the skin moves. And because the software is deforming the geometry, it’s a lot more faithful to the actor’s performance. For instance, when the actor is talking and the English phonemes reverberate through his face, the software picks that up really well. And that’s why the dialogue hit in the right places as opposed to doing key frame animation, in which you’re basically making up the way the dialogue is delivered.
Helman tested the technology by recreating a scene from Goodfellas – the one with the pink Cadillac. According to Helman, the Irishman star loved that he didn’t have any markers on his face. Once Martin Scorsese gave his approval, everything fell into place