VFX supervisor Richard Clegg raises the bar on photorealistic digital doubles in Denis Villeneuve's hit follow-up to the 1982 sci-fi classic directed by Ridley Scott.
For MPC, the project started not with script pages, concept art or live-action plates, but with director Denis Villeneuve’s imperative: “This cannot be discussed with anyone.” So began the super-secret work on the digital recreation of actress Sean Young’s “Rachael,” transported through time from Ridley Scott’s 1982 sci-fi noir classic Blade Runner to the just released sequel, Blade Runner 2049.
Though tasked with only 16 Rachael shots, of which a mere half, eight, showed her face, MPC VFX supervisor Richard Clegg’s and his team were handed the centerpiece of arguably the most poignant and gut-wrenching scene in the entire film -- not just your normally difficult photoreal digital double work, but the recreation of Rachael, the stylishly gorgeous replicant whose escape with Harrison Ford’s “Deckard,” in the original movie provides the foundation upon which the new film is built.
But absolute secrecy was the least of the project’s challenges.
As Clegg describes, “Doing a photorealistic, fully digital person is still quite a challenge and a very difficult thing to get right. People, subconsciously, are experts at what faces look like. We constantly interact with other people, and even subtle shifts in a face, the way people move, can change how you feel about them. We communicate a lot with subconscious body language as well. So, if you don't get it 100 percent right, if you don't nail it, if there's something off, something wrong, it's a fail. You can't do an ‘OK’ version or get it 80 percent right. It's got to be spot on.”
Working on set with Ford and British actress Lauren Peta, who stood is as Rachael’s body double, Clegg quickly realized how critical MPC’s work would be to the emotionally charged scene. Notes Clegg, “When we were shooting, I was watching Harrison Ford and Lauren Peta’s performance and thought, ‘This is a highly emotional scene.’ It's one thing to build a realistic looking head, and it's another thing to then have it move. But, it's extremely difficult to make you feel empathy for that digital person. It needs to feel like an entity, a being that has a soul, rather than just a nice looking picture. So, the stakes [for creating a digital Rachael] were very high.”
Clegg and his team worked with a combination of historical reference materials, on-set data capture, and full 3D facial scans as a base for their animation. “Obviously, the main target was [Rachael from] the original movie,” he explains. “Especially, that opening scene where she walks into Tyrell's office. Our goal was to make an exact digital replica of that person. But, the film's stock is quite old, the footage is old, it's slightly soft, it's grainy, and it's dark. There was harsh, contrasting lighting. Additionally, she had distinctive makeup, which tricks the eyes as to what’s shadow and form on her face as opposed to what’s lighting. So, we needed to use other material to fill in the gaps, to assist us in building the digital head.”
Those other materials included footage of Young from other films, including her role as Chani in Dune (1984), as well as photographs of her from that same time period. Young’s head was also scanned to provide a solid base of 3D data from which Clegg’s team could build the initial skull upon which the younger head could be made.
According to Clegg, the main build started with the live-action Peta footage. “We had Lauren perform the take dressed in full costume. We used her body, shoulders, and legs, but everything from the neck up, we fully replaced digitally. On the walk-in shot, we did do some distortions to the photography. We narrowed her shoulders and did a few tweaks to bring it into line with the original movie.
“Once we had footage, we matchmoved it, tracked and created our digital cameras, built the stage and entire set in 3D,” Clegg continues. “Then, we built a digital version of Lauren, just her head and body, so that we could track and match the movement of the head exactly right. We call that roto-animating. So, basically, from there, we have a digital version of Lauren. It's not something that we rendered or made any pictures with. It was just a 3D representation of her that's moving around in the correct space.”
From that “digital” Lauren came the digital Rachael. Says Clegg, “Once we had this full shot, we built our digital Rachael head, swapped out Lauren, popped in Sean, then rendered and composited everything.”
If you think that sounds quite simple, you’re mistaken. With only a small number of old photos, some film footage, and no camera, lens or other data normally captured with VFX reference photography, Clegg relied heavily on the present-day Young head scan. “We had a very precise scan of her head as it looks today and from that, we built the underlying skull anatomy,” he notes. “Lots of things on your face change over time. But the skull is something that stays relatively fixed. From that skull, we could get the key bone structure, like around the eye socket, the bridge of the nose, the jaw and so forth. Once we had that skull, then we could start to position and line up the head against shots from the original movie. Our artists were able to go in and model, sculpt and build her head so that it was a perfect match.”
The next step was convincing the director as well as overall VFX supervisor John Nelson that the match was indeed perfect. But how? For Clegg, that meant a visual test. He explains, “Obviously, we needed to convince Denis, John, the producers, everybody, that we'd nailed the head with an exact match. To do that, we recreated three shots from the original movie, where Rachael has a more subdued expression, where her face wasn't doing too much. We tracked her, introduced a little subtle animation and recreated those three shots. Then I cut them into the original movie. I took a short clip and cut in a digital version of one shot and asked them to guess which one was the new CG shot. In the end, they guessed it. But, it took them a little while. That’s how we knew we’ve got it. There were high-fives all around, fists pumping in the air. That was a good day.”
While Rachael’s look had to be spot on, the believability of her performance was even more critical. Even with extensive 3D facial captures of both Young and Peta, done one day after principal photography was completed, where the director recorded the subtleties of their facial expressions, the way their skin moved, their eyes twitched, their nostrils flared during various extreme poses, in the end, Clegg’s team had to completely animate Rachael’s performance by hand.
“Initially the plan was to use this videogrammetry [facial capture] data to drive our digital head,” Clegg describes. “But ultimately, based on performance requirements and keeping the likeness of an early 1980s Sean Young, we animated the whole thing by hand. That's actually something I'm quite proud. She is all hand-crafted. The face is fully animated and puppeteered by people. There's no data capture that's thrown in there. That how we achieved such a high level of emotion. Animators could keyframe every little nuance and slide on the skin, every twitch in the lips. Even the ears are animated.”
The other critical asset on this project was time. Clegg’s team was given 10 months to deliver the work. “We shot in September [2017],” he notes. “We started working in October and finished the final days of July.” They spread the work over three MPC studios: Montreal, London and Bangalore. Says Clegg, “Most of the assets were built in London with a bit of support from Bangalore. That work was supervised by Axel Akesson, our asset supervisor. He led all the look development, modeling and the texturing. I think they had a team of about four or five people. There were 10 of us in Montreal.”
Clegg went on to explain that while his team used their standard bag of pipeline tools, they used them to their fullest. “There was nothing special we used, just our standard toolset,” he says. “But, we certainly had to hit an incredibly high level of output with them.” For modeling and texturing, they used Autodesk Maya and Foundry’s Mari. For animating and 3D work, their core pipeline tool was Maya. For lighting, they used Foundry’s Katana, and Nuke was employed for compositing, with Pixar’s RenderMan used for rendering.
“We have a couple of proprietary softwares thrown in there too,” Clegg adds. “All her hair was done with an in-house software called Furtility. We also obviously have a few little bits and bobs and tools that we write ourselves on top of everything else.”
Ultimately, the challenge Clegg faced was not just creating a digital double, but a fully digital character that could convincingly share the screen with Harrison Ford. “Rachael is such an iconic character,” he insists. “Her performance is cut in with Harrison Ford, a top actor giving an emotional performance where he's tearing up. We had to nail it for people to buy it. Because, if we just cut to him looking like he’s talking to a vacant doll head, it would throw everyone out of the scene. The biggest challenge was giving her soul. Especially in her eyes. All the nuances and head movements. That was the hardest part -- making sure her performance was top-notch.”
Dan Sarto is Publisher and Editor-in-Chief of Animation World Network.
