The Two Towers: Face to Face With Gollum

Andy Serkis provided the voice and acting for Gollum, which the animators used as a foundation in delivering the character's final performance. All images © 2002 New Line Productions. All rights reserved.

In The Two Towers, a favorite scene of audiences was the schizophrenic argument of the submissive, frightened Smeagol with his belligerently self-protective alter-ego Gollum. With the two personalities cleverly juxtaposed, we see in a moment the wonderful, expressive range of actor Andy Serkis and the team of Weta artists who successfully brought the performance to life.

Gollum is, without question, one of the highlights of The Two Towers, and considered to be among the best performances of the film. It is no mean feat for one character to arouse so much contempt and pity. Yet we hold our sympathy for Gollum somewhat at arm's length; it is a hesitant trust. We want to like him, but don't know quite what to make of him.

A Host of People and Techniques

With the motion and voice originally supplied by Serkis, the final on-screen presence was crafted and honed by a team of 18 animators. Bay Raitt, the creature facial lead for the movie, likes to use the following analogy: "Andy Serkis wrote the music, then [animation director] Randy Cook conducted it, and the animators are the ones playing the music that you see in the film."

Gollum was created through a combination of techniques. To begin, Serkis would be on set, shooting Gollum's scenes with the live actors. Serkis' movements would be match-moved as a rough performance onto a digital puppet for the animators to use, so they could begin blocking in the face and the beats. Then, Peter Jackson would decide that the performance could be pushed a little more, and Serkis would come in again to do his scenes on a motion-capture set.

Motion-capture data oftentimes has all sorts of problems, because it doesn't capture faces, hands or feet very well. So, before the data would get to the animators, the motion editing department would fix the feet and hands, and all of the pops and jitters. Then, given video reference, the animators would start animating the fingers and hands, the feet and face, on top of the motion-capture. Sometimes, in this process, new dialogue for Gollum would come up, and the animators would start over again. In sections, they would get direction from Jackson that certain aspects of the performance look good, while other aspects could be completely reinvented. For example, instead of Gollum looking scared in one moment, perhaps he would be called upon to look tough. The animator would then take a section of the animation, delete Serkis' performance, and re-keyframe it. This could be for certain body parts, or just the head and neck, or the whole character. Raitt explains, "So, when you watch the actual sequence, it's not motion-capture, it's not motion editing, it's not keyframing, it's not rotoscoping...It's a mishmash of all of the above."

A lot of congratulations are rightfully showered upon Andy Serkis. Serkis was as involved as any of the artists in breathing life into Gollum's character. When requested, he would even come to the animators' desks to act out a scene for them, so the animators could make their own video reference of his performance.

Raitt comments, "Andy definitely deserves the applause, as do the animators. They took the data as a starting point, embellished it, sweetened it, and made sure the intent of the performance was really clear. Sometimes the footage was so blurry or bad, or the direction had changed, that it was entirely keyframing. It is important to note that there is no facial motion-capture data, at all, on Gollum. The only motion-capture data is for his torso, legs and arms."

For Gollum's range of expressions, Raitt and his team created a system of sculpted faces for the animators to use. The animators would, almost like playing a musical instrument, use the facial system to bring Gollum's emotional world into being. Among the lead animators for Gollum were Adam Valdez, Atsushi Sato and Mike Stevens. While the animators definitely took a lead from Serkis, from an acting and presence point of view, as Raitt explains, "At the end of the day the animators were responsible for delivering the shot as a final piece of performance."

Facing the Music

What exactly does the job of a "creature facial lead" entail? Raitt was responsible for delivering to animators a facial puppet that they could animate, and that would show up as a reasonable approximation of what they would eventually see on screen. Raitt and his team, along with the creature department, were responsible for delivering, as he says, "Gollum from the neck up."

Raitt explains, "Gollum's head began as a cube on my screen. Using scan data I got from the [Weta] Workshop, and working back and forth with the animators and Randy Cook and Peter Jackson, I modeled his face, laid out the UV's [texture setup], and then delivered that puppet to the animators so they could get their shots done and out the door." Minus the textures and shading, Raitt sculpted the 875 combination shapes that are present in the articulated face of Gollum for the final film. "I've basically been staring at Andy Serkis and Gollum for the better part of three-and-a-half years," Raitt says.

Originally from northern California, Raitt did not go to university, but rather started working for Image, coloring comics and doing "the occasional back pin-up" for some of the early issues of Spawn, The Pitt, The Maxx and Akira. Moving his way south to San Francisco, Raitt then took up residence at Protozoa, doing digital puppetry and performance animation for video games and television. Eventually, he moved down to Los Angeles to help as a product manager for a 3D software system called Mirai created, at the time, by Nichimen Graphics, but which has since gone into the hands of a company called Izware. From there, Raitt moved on to Weta, Ltd. in New Zealand to work on The Lord of the Rings trilogy. "It's kind of an odd career path," he summarizes.

When he arrived at Weta, Raitt was charged with designing and building the system they would use to sculpt the faces of the trilogy's hero creatures, including the Balrog, Cave Troll and Treebeard. After surveying what tools were available, and doing some prototypes, Raitt decided the best way to define the characters would be to sculpt combination shapes for their expressions. Mirai, a 3D sculpting engine that has been around for about 17 years, was the tool of choice.

Animator Jason Schleifer, who used to work at Alias|Wavefront, came on board the Weta team at about the same time as Raitt, and the two of them served as the essential support team in building the digital puppets for Gollum. While Raitt concentrated on modeling and sculpting Gollum's head, Schleifer built the puppet and rigs that control Gollum's body. Raitt and Schleifer wrote the code that brings together the individual digital models. When saved out to a directory, the system automatically compiles the models, so that the sculptures can be moved around with a set of sliders to see how they work together. Raitt says, "Though Mirai has a little better tool set for sculpting, the animators can interact with, and manipulate, the puppets in Maya."

In traditional animation, of course, the lead animator lays down the keys and the assistant animator creates the inbetweens. What Raitt has accomplished with his combination sculpting, essentially, is to lay out the "keyframes" for Gollum's face. The difference, however, is that instead of creating keyframes that are laid out sequentially in time, the combination sculptures are keyframes that have to look good shuffled in any order, and from every angle.

The on-set performance of Andy Serkis (top), and the animated digital performance of Gollum (bottom).

In the three years he has been working on The Lord of the Rings, Raitt has sculpted over 10,000 shapes, or facial poses. He explains, "This is what Gollum looks like when he's frowning. This is what he looks like when his eyebrows are up. But this is what he looks like when his eyebrows are up and he's frowning." Using a set of 64 controls, in combination with one another with names like "Lip Up Raiser," "Lip Low Depressor," "Brow In Raiser," "Brow Out Raiser," "Nose Wrinkle," etc. the animator has control of moving among the sculpted shapes to craft a character's performance.

While Raitt worked mostly on sculpting Gollum's facial expression range, the same system and methodology was used by creature technical directors John Feather, Sven Jensen and Brad deCaussin to create the other characters such as Treebeard. While the design for Gollum was being approved for the The Two Towers, Raitt was looking throughout the industry for people who could help build the characters as quickly and expertly as possible. Raitt remembers, "I grabbed these guys because they were really good digital sculptors, they knew Mirai, and because I could sit down with them, during training, and pour into their capable hands the two years worth of experience I had with Gollum."

Creating a host of shapes that can all play together nicely, under the control of the animators, is a kind of artform which Raitt refers to simply as "combination sculpting." While Raitt himself created roughly 75-80% of Gollum's face, he would assign to his team other sections of the face to sculpt. "For example," Raitt says, "when Gollum sputters his lips, blowing air out of his cheeks, those are Brad's shapes. Gollum also has all of this beautiful, sexy stuff that happens with his skull and the back of his jaw when he opens his mouth, or when he says, 'Oooh' John Feather sculpted a bunch of those shapes. And when Gollum sort of squints his eyes, and his lips stick together, Sven sculpted those." All of the facial models filtered through Raitt, and he massaged the sculptures into the larger shape network. The team had to sculpt as fast as they could, even up to the end of production. Raitt says, "Almost three weeks before the last shot was finalized, I was still compiling new faces for animators for specific shots."

Back to the Drawing Board

Following The Fellowship of the Ring, Peter Jackson decided the original design for Gollum didn't work as well as he wanted, in lining him up next to Sam (Sean Astin) and Frodo (Elijah Wood). Once Andy Serkis was cast to play the part of Gollum, Jackson even considered for a time putting Serkis directly in the film.

Raitt explains, "The original design didn't work, per se, because it was a little more creature-oriented." Mike Asquith, who had sculpted the design maquette of Gollum for the first film, had done "a lot of beautiful stuff with skin, just hanging right over the bone, without any muscle," Raitt says. "But when I tried pulling that around [digitally], it became a little bit difficult, because it looked like the skin was moving for no reason at all."

At the time Asquith had sculpted the original Gollum, Andy Serkis hadn't been cast yet. Raitt continues, "Mike did some initial character studies, which helped to define a lot of the feel for Gollum these small head busts, just some beautiful stuff. They don't look exactly like Gollum does in the movie, but, from a sculpting pass, they were the original inspiration. We were like, 'That's it. That's what we want.'"

When Serkis was cast, it was fortuitous because, as Raitt says, "He really looks like this design maquette...though we knew we could probably make it look even more so. When Andy first saw it, he said, 'Christ, that looks like my dad.'"

The idea for the redesign stemmed from a drawing by Christian Rivers. He took a snapshot of the Gollum model from the first film, a photograph of Andy Serkis, and drew a sketch mixing elements of the two. From that single sketch, Peter Jackson realized they could tune the original Gollum to look more like Serkis, and so they decided to go ahead with a redesign.

Starting from scratch, Peter Jackson, Richard Taylor, Christian Rivers, Alan Lee and Jamie Beswarick all helped in the intense design phase to re-imagine Gollum.

Raitt says, "Jamie is one of the mad geniuses at Weta, who physically sculpted the Cave Troll and the Nazgul's steeds, Gollum's body and face, and the Orcs. He's the best sculptor on earth in my opinion."

Raitt and Beswarick collaborated closely in bringing the new Gollum to life. Raitt explains, "Jamie would do a bunch of sculptures, and we would talk about them, and I would suggest things that might not move very well. Back and forth, we would work. I would grab a sculpture off his desk, drag it back into our building...The guys in the modeling department would scan it, so we could have rough data for it...Then I would build a digital model of Gollum's face from a cube, and try to line it up to the scan as best as I could. Jamie would come and sit at my desk, and we would pull the model around, see how it moved, see what it looked like with the brows up, and when it was smiling. As we tested the design, Jamie would see, and say, 'Oh, we need to change this, and this.' Then he would run back to his desk, and sculpt the changes in clay. Back and forth, back and forth, until eventually Peter approved it."

Reason to Smile

Having worked with Gollum for so long, Raitt says, "It's kind of weird to be staring at the same thing for three years, even if it's a piece of moving sculpture that the world seems to like, and that you've had a childhood affinity for."

Working on Gollum's smile was an initial concern for Raitt. "That kept me up more than anything else," he says. "If we screwed up Gollum's smile, the whole Fellowship, the whole movie, wouldn't really make sense to me. The only reason The Lord of the Rings really happens is because Gollum is pitiful, and people haven't killed him many times over. If his smile rang false, if it didn't strike the right chord, you wouldn't buy into him as a character."

Raitt did hundreds of versions of Gollum's smile. Everybody had an opinion. "I remember that there were a few people that I would bump into who actually had Gollum's smile," Raitt recalls, with a chuckle. "I would sit there and try to not be overtly looking at them." He admits, "I got really good at having a little piece of friendly chit-chat with people, while also being able to absorb what their face does when they move, which is a really disturbing skill to have acquired."

Raitt adds, "I used to go to movies and sit in the front row, pretty much ignoring the film, and I would just watch the faces, just to get an idea of how big Gollum's face was going to need to play and how flexible it needed to be. For the first year-and-a-half I was here, I would walk out of films and feel queasy and sick with angst and worry, and that I was somehow going to be responsible for playing a part in screwing up this beloved character. But something happened, we figured it out and came up with a really good system. The face started to just sink in, it was flexible, it looked good, and we could make changes quickly."

Raitt is happy with how the facial system has come together, though it has been an amusing challenge to keep the characters consistent. One of the more common tendencies for animators is to animate their own faces, in part because it's one of the best references they have while working. Raitt remembers, "Some of the animators, brilliant animators, would have this facial twitch, or a quirk of their own features and expressions which was charming on them, but which they would want on Gollum as well. I have had to integrate all of these personalities, and the input of these exceedingly talented people."

Raitt concludes with an anecdote from TheOneRing.net, posted by a woman whose husband is hearing-impaired. The man is always frustrated when watching animation, because he can not understand what the characters are saying. The lip-synch is never quite right, or not good enough. However, when he walked out of The Two Towers, he turned to his wife and remarked that it was the first time he has been able to lip-read a computer-animated character.

Greg Singer is an animation welfare advocate, eating in Los Angeles.