Electric Dreams: John Nelson Talks ‘Blade Runner 2049’

‘Blade Runner 2049’ © 2017. All images courtesy of Warner Bros.

With the Oscars just two short days away, we conclude our in-depth feature coverage here at AWN of the best visual effects nominees with an intimate and revealing interview of John Nelson, overall VFX supervisor on Denis Villeneuve’s gritty and grey Blade Runner 2049, the long-awaited sequel to Ridley Scott’s 1982 sci-fi noir classic, Blade Runner. Nelson was responsible for almost 1,200 visual effects shots, spread across eight VFX houses -- he spent two full years on the film, beginning with the early stages of pre-production right up to final post, working intimately with production designer Dennis Gassner and legendary cinematographer Roger Deakins (nominated for the best cinematography Oscar a whopping 13 times, including this film) to bring to the screen Villeneuve’s dystopian vision of a dying world and the dangerous secret that could bring about its final demise.

AWN had a chance to sit with Nelson late last year at the 2017 VIEW Conference in Turin, Italy, where he spoke at length about the film’s main challenges, which included the never-ending, smog choked Los Angeles cityscapes, Joi, our hero blade runner K’s digital holographic companion, and the recreation of Sean Young’s iconic Rachael, the beautiful replicant whose story tragically binds both films together.

AWN: You were on this film from the beginning. Where did you focus most of your attention?

John Nelson: My job was the overall lead supervisor that is on the movie from the beginning to the end, from pre-pro [pre-production], through principal photography and then through post [post-production]. I did everything from analyzing the script to breaking it down into how many shots we’d have and how they’d be done -- then I worked on designing the shots, the photography of the plates that we used to make the shots and then, on the actual posting with the visual effects houses that I hired and the visual effects supervisors that worked at those houses that I trusted to complete the vision that the director told me he wanted.

John Nelson, overall VFX supervisor for ‘Blade Runner 2049.’ Image courtesy of VIEW Conference 2017.

What I really try to do in every instance is…I’ll listen to the director and try to give him three choices that are all around what he’s asked for. Different variances. And then hopefully, he’ll choose one. Once he does that, we’ll delve deeper into that choice to try and make it even better and more special.

AWN: How much did you take into account the visual design and tone from the original movie? Did you start with any concept art? How did you arrive at the “look” you wanted?

JN: Denis definitely knew what he wanted. He comes from Montreal. He said, “In my Blade Runner, it’s going to be more like Montreal, because that’s my background. So, it’s going to be snowy in some places. Instead of dark, in some places it could be a little lighter.” We all took a look at the original movie…I mean, everybody loves the original movie. So, we took a look at everything from the original movie, down to the Mentor Huebner [a production and concept illustrator on the original film] designs, to the Syd Mead [a designer and visual futurist on both the original and new films] work, all of that [original design] stuff. Then we took a look at a lot of different concept and comic book art that might be appropriate.

Denis had been working with Dennis Gassner [the film’s production designer] by the time I got on the movie. They had been doing concept art. So, there was a lot of stuff that Denis would find and say, “I really love this, it should be like this.”

For the cities, Denis and Roger [Deakins, the film’s director of photography] had a picture of the favelas in Mexico City. He described that there’s one big city all the way from San Francisco to Los Angeles…there really isn’t a lot of negative space in between. When you’re way out in rural places where Sapper’s farm is, that is real. But, when you come onto the Grapevine [a graded highway that leads up and through the Tejon Pass and Tehachapi Mountains north of LA], it’s like solid city. I looked at that and said, “Well, let’s see. Why don’t we shoot some plates in Mexico City and double Mexico City?” I searched on Google Earth for all the places in Mexico City that I liked. I did flyovers, complete with GPS coordinates, then passed that off to our location scout and aerial camera man, Dylan Goss -- he had worked with Roger before…Roger trusted him. Roger would tell him how he’d like to shoot, which, ultimately, was always overcast.

The endless Trash Mesa of ‘Blade Runner 2049.’

Any exterior that we shot, we always shot overcast…no hard light…because we’re trying to simulate soft light, and if you shoot plates in hard light, there’s no way to take the hard light away. In all the plates we shot in Budapest, we’d always wait for [when it was] overcast. In Mexico City, any place that we would shoot, we would always research the weather patterns and ask, “What’s the cloudiest month?”

We looked at the best concept art from the original movie, but it wasn’t like we were in lockstep. For two years, I dealt with everyone and their brother coming up to me and saying, “It’s my favorite movie. Don’t mess it up.” Everyone that worked on the [original] movie loves it. We all realized early on that we were going to make a different movie. But, you had to honor the first movie -- it had to feel like it’s the same world.

The first movie is sort of a future noir picture…there are elements of film noir. There are elements of things that are old along with things that are futuristic. We tried to extend that in the new movie. I don’t know if anybody noticed, but in Blade Runner 2049, there are no 16x9 screens, it’s all 3x4s. It’s all like old 3x4 screens. And there are no cell phones. It’s this alternate reality that goes from the end of the first movie and extends another 50 years. Everybody was into that.

There’s this whole backstory…what happens in the time between the first movie and our new movie. Some of it’s in the prologue and some of it is in these short films, like Watanabe’s film [celebrated anime director Shinichiro Watanabe]…Alcon paid several filmmakers, like Luke Scott, Ridley’s son, to produce [three short films]. Luke did a couple of the shorts [Nexus Dawn and 2048: Nowhere to Run], and then Watanabe, who did Cowboy Bebop…he’s a brilliant anime guy…he did an anime version [Blade Runner: Black Out 2022] and they sort of explained the time between the old and new movies.

It was an experimentation process, both through the art department and visual effects. When we were in pre-pro, we’d say, “What’s good from the old movie? What’s good from the old Mentor Huebner and Syd Mead designs…the scraps that Denis and Roger would pull together?”

The futuristic Los Angeles of ‘Blade Runner 2049’ is choked with atmospherics like smog and rain.

AWN: The original Blade Runner was distinctly lit, with a grainy, almost blurry darkness. Nothing bright and shiny. How did you, Denis and Roger approach the lighting on the new film?

JN: Well, that’s really a question more for Roger than for me. Roger would say that he’s different than Jordan Cronenweth [director of photography on the original Blade Runner]. That’s what he would tell me. Jordan lit in a certain way, but this is another movie. The first movie, there were a lot of spotlights. And for our movie, one of the main premises I was told by Roger and Denis is that the movie is atmospheric, in that the weather’s changed so much that there’s an atmosphere like Beijing, only 100 times worse. So, everything is soft. We Google searched and found places that use a lot of coal in China, where the atmosphere’s just completely smogged up. We all loved that because it would give you this gray landscape where the colors would pop.

I got references from both Roger and Denis that we locked into for out atmospheric tests. We would actually run atmospheric tests on all of our stuff, whether it was going to be total or partial CG. We would take a shot, then projection map it onto digital surrogates and actually run volumes. “What if the atmosphere ended here? What if it ended here? What if it ended here?” We had a strategy, because we wanted to be so atmospheric that it would be really beautiful but claustrophobic. Denis wanted it to feel claustrophobic. But, in the wide shots, we needed to cheat a little bit, because if we kept the same atmosphere for the close ups that we had in the wide shots, you wouldn’t see 10 feet. So, you were constantly balancing [between the two]. Like I say, Roger defined the lighting, that’s for sure. I come out of camera myself, so what I always try to do, especially with a brilliant cameraman like Roger, is just follow their lead.

I’ve done a lot of CG and a lot of bluescreen. We all agreed, early on in the storyboarding and breakdowns, of how we were shooting these big visual effects scenes -- and there are lots of them…there are 1,190 visual effects shots in this movie, almost 1,200 -- we all agreed that to make it feel right, around camera and around the actors, we wanted real sets, to have them to be interactive with real things. Now, that being said, every wide shot in the movie is a visual effects shot, because you just can’t to afford to build the city, right? Or build the trash mesa that goes on forever.

I would follow Roger’s lead on lighting, look at all the design scraps of Beijing atmosphere, and try to match into that, which we did fairly effectively. Later, when Roger came in to time the movie, we would have big review sessions with Skype where Denis, Roger and I would all be looking at the shots together and giving comments down to the vendors. It was a collaborative process, but I was definitely following Roger’s lead.

Las Vegas in ‘Blade Runner 2049.’

AWN: Describe the dynamic of deciding what, and how much, to shoot in live-action.

JN: From Dennis Gassner, the art department’s point of view was, we had sets, we had a whole studio in Budapest, we would build as much set as we could. We had big sets, but they weren’t endless. We had a backlot for Sapper’s farm and the trash mesa. It was probably as big as a football field. So, half of it was trash mesa and half of it was Sapper’s farm. Half a football field is big, but it’s not that big. We based the trash mesa loosely on Chittagong, Bangladesh, where the boat graveyards are. We were going to shoot there, but it just got too hot for foreigners. I sent a local man from Asia over to scout it both with stills and video. I was worried that it might be an issue, actually shooting plates there, because politically, it was getting very hot. He actually did photogrammetry passes at the same time he did the scout. Those are the passes we went back to generate the geo with.

What would be CG? Dennis, Roger, myself and Denis all saying, “Well, it makes sense to build out to here. Then the rest has to be CG.” But, from Roger’s point of view and my own as well, if you can shoot it, we should shoot as much of it as we can. We should build as much as we can. But for the big city stuff, forget it. I think we shot on the streets of Budapest maybe one night. Because, what we’re making really doesn’t exist. So, let’s get what we can. And then we had this huge build in CG for LA and Vegas.

AWN: The scenes with digital Rachael were an absolutely gut wrenching and emotional part of the film. Her appearance in the film was kept completely secret. The digital recreation was hugely difficult. Were you worried that she might not look good enough to use? Was that ever in doubt? Or did you figure, “This just needs to work because she needs to be in the film.”

JN: It was pretty much, “This just needs to work because she needs to be in the film.” We knew, early on, the hardest things in the movie were the cities, Joi and Rachael, who we had a code name for…Rita. It was super-secret. Nobody could talk about it at all. I wanted it to be on the cover of Cinefex and we couldn’t even get them [the studio] to approve a picture for it because it was so super-secret.

I did this huge analysis of all the digital humans that have been done to date, from the ones that worked to the ones that didn’t. The great thing is, everybody lives on the shoulders of everybody that’s come before. You can take a look at what’s been done and say, “Well that really worked,” or, “That had issues.” I knew then only several people in the world could do it well. We ended up making a deal with MPC, who had the done the digital Arnold [from Terminator: Genisys]. I also really liked the work that Digital Domain had done on Benjamin Button. It’s one thing to make a digital human look real. It’s another thing completely to make them act and emote emotionally correct.

Rachel is re-created in CG for ‘Blade Runner 2049.’

We knew we would get Sean [Young, the actress who played Rachael] and actually use her. We found an old life cast of Sean when she was 28. She was 19 when she did Blade Runner, and really, it doesn’t sound like a big difference. But, there was a difference. That life cast got us in the ballpark. But, the biggest thing that really helped us was when we started analyzing Sean’s mannerisms from the original Blade Runner. Rachael is different than Sean. Sean would leave her mouth open. Ridley [Scott, the director of Blade Runner] would go, “Close your mouth. Keep it closed.”

Sean in Blade Runner is just about one of the most beautiful women I’ve ever seen in my life. She walks up in the original movie and Deckard says, “Is that all artificial?” She goes, “Yes.” “Is it expensive?” She goes, “Very. I’m Rachael.” When she walks up, she’s confident and smart…she’s like this beautiful nerd. Sean had these mannerisms, like when Tyrell says, “I want to see it fail. Try it on her,” and she does this little wry smile and bobbles her head. I talked to Sean about it. She said, “Yeah, my head bobble,” and I go, “Yeah, exactly.”

We drilled into all those mannerisms from the original movie. When we were modeling the head, we’d show the model in a turntable, and the producers and Denis said, “Well, it’s close to Sean Young, but it’s not Sean Young yet.” I said, “Okay,” went back and worked it even further. Then what I did was I took three scenes from the original Blade Runner, and I replaced the head in one shot in each of them. Then I showed it to the producers. They said, “Why are we looking at this? This is the old movie.” I said, “Because one of the shots in all three sequences is the digital double,” and they went, “John, you bastard,” like, “You should have told us.” I said, “Well, no. I wanted to see if it would pass the test,” and it did. At that point, we knew the model was good. But, we needed to get the emotion right.

We cast this woman, Loren Peta, who Denis chose to be the Rachael double. She looked very good, a little more athletic than Rachael. We put her in costume and hair and dotted her face. She acted the whole scene with Harrison [Ford, the actor who plays Rick Deckard]. Sean Young was on set those days as an advisor. He directed Loren first. Then we did a DI4D facial motion-capture, with Sean and Loren, on a super-secret Saturday, shrouded in secrecy…Sean and Loren saying the lines. Then, ultimately, we put the head replacement into the scene.

It was imperative that, when Sean walks up, she really has to have that same confident gait of Rachael in the original movie, like, “That’s the Rachael I remember.” Denis said, “Okay, John, so once she gets up, it’s like two people who are in love and haven’t seen each other, and they see each other at a train station, like after 15 years, and the emotion just flows out of them. It’s hard for them to control it.” She walks up confident. Then she goes into longing. Then she goes into rejection. You know what I mean? That’s the emotional compendium of the performance. We went back to the original Blade Runner and tried to find those moments, really drill into those things with the eyes, and we put those emotions into the performance.

Apart from that, we had a brand-new subsurface scattering renderer, which was great. We really drilled into makeup. We asked all the women on the crew to explain makeup to us. We did all this different sampling of eyeballs. Sean’s eyes were really different than a normal person. She has these huge eyes, and they jet out a little more. Of course, the eyes are the windows into the soul. We’d take whole Saturdays just dedicated to every shot in those sequences…what’s the nuance that we’re trying to drill into, particularly for Joi, for the threesome with Joi and Mariette and with Rachael, because we really needed to get those right.

Ana de Armas (Joi) and Mackenzie Davis (Mariette) are joined in ‘Blade Runner 2049.’

AWN: Looking back, were there things you thought would be tough that weren’t as difficult as you expected, and vice versa, were there things you figured would be pretty simple that proved unusually difficult?

JN: All the scenes were really hard. I don’t know if things were harder than I expected, because I expected them to be really hard. One of the reasons we had eight vendors is I kept the shot count down, so everyone could have really special work that’s pertinent to them. We had tons of effects. The movie is two hours and 43 minutes long, and there is like an hour and 40 minutes of visual effects. I mean it’s a lot…it was the cities, Joi, and Rachael…we had Vegas, Sinatra, Elvis, Ana’s lab…Ana’s lab was a huge science project because we had to shoot kids, and we had to control the space and time of the kids. Ana could reverse speed, and that all hadn’t been worked out before we shot it, except that I knew I had to control it.

We storyboarded the whole movie, so Roger had set where we would put the camera. We would have Ana…even though we were going to shoot her later when her set was there…we shot the kids on bluescreen, we’d line up Ana and the kids as if it were a normal shot, then we’d pull out the Ana stand-in, and we’d shoot the kids. But, before we’d do that, I’d look at the tape measure from the kids to Roger’s camera, then draw a semicircle on the stage. I put two cameras here, two cameras here, and Roger’s camera. Then we brought in the kids, and I filmed the whole action with five cameras. Then, afterwards, we shot Ana again, on another set with interactive light, the same interactive light we shot with the kids. All of the movement of that five-camera array, that’s all morphed so we can control them…go forward, backward, stop…okay, reverse.

Everything on this film was hard. But, the hardest stuff was definitely the cities and trying to make Joi come on correctly.

But really, the hardest stuff was always the cities. Some of those were extensions. Most of them were full digital cities. Then, Joi, making Joi come on correctly, her back shell, and Rachael. Bringing Rachael back. Rachael reborn.

With Joi, one of the big things we had to break through was what we now call the back shell. We experimented with holograms -- making her seem like a normal hologram -- which we didn’t like. We didn’t like too much sparkly stuff, because we wanted the film to feel very analog and not digital. We had to come up with this back-facing volume for Joi. What I mean by that, when you look at a glass of water, you see the front part of the glass, then you see the back part of the glass. But the back part of the glass is backwards if you were looking through it. What we did was, when we shot Ana [de Armas, who played Joi] doing her performance, we had witness cameras all around the room. Then we would roto the digital surrogate for Ana, frame for frame, completely right on from multiple cameras. Not only Roger’s camera, but all the witness cameras. Then, we would map Ana onto the front part of that geometry. In CG, we would make the back part of the geometry. When you looked at her, she would be photoreal, because she was photographed. But, transparency on her body would show a volume. When she pivoted, the front part of her would rotate one way, but the back part would rotate another way, because that’s really what it would do. That was very hard to get right and balance.

Making the two women [Joi and Mariette] merge, that was really hard. Certainly, Rachael reborn. Getting the digital head. The nuances of the makeup and the performance, the verisimilitude of getting a little tear in the eye, all of that stuff.

There was this passion on the part of everyone on the movie. It all stemmed from Denis, who was massively passionate about it, and Roger as well. Everybody wanted every shot and every sequence to be above that bar. Everybody brought that passion to the work.

I worked for two years, literally. You get so close to it. It’s good, but you just have to…you’re literally so immersed in it, it’s almost like a virtual reality of your world. In fact, for the first two weeks after the movie was over. I would walk around and go, “Hey, was that building here before?” You know what I mean? People are out having dinner. Like wow…what a concept. It took me a while to actually relax, because it just was this massive, massive project.

On a movie like this when you have a great director, great actors, a great script, a great cinematographer and a great editor, things become exponentially better, because everyone does their part. Great films come from great directors. Denis is absolutely a great director and a wonderful human being at the same time.

Dan Sarto is Publisher and Editor-in-Chief of Animation World Network.