MoCap Makes Realtime Previs A Reality

In today's world of high-tech production, realtime, onset previs is becoming a reality. Just how important is it? If you're familiar with ILM's Zviz, you know that George Lucas believes this interactive previs system utilizing the LucasArts game engine will eventually revolutionize the industry.

Meanwhile, MoCap is becoming more integral to previs, and a major pioneer and the leading developer of optical motion capture solutions is U.K.-based Vicon Motion Systems. In 2004, Vicon acquired Los Angeles-based House of Moves, the longest-standing independent motion capture service bureau in the world. The result: Vicon House of Moves has set up three permanent motion capture stages. Each one is capable of providing realtime previs for their clients to use in a variety of ways.

"We have to remain flexible to enable us to customize workflow on and off set to meet the individual needs of clients and their productions," explains Gary Roberts, President of Vicon House of Moves and the Vicon Feature Unit. "Every production we shoot at HOM, whether it is a feature, commercial, music video, videogame, TV show on or off location always uses on set realtime previs. It is incredibly beneficial for both our clients and in house staff."

Their main stage utilizes more than 70 Vicon MX F40 cameras to capture full body and finger motions along with props, cameras and set pieces in a large capture volume 90'x45'x18' stage. Their second stage has a slightly smaller capture volume but is optimized to use more than 120 MX F40 cameras to capture the performance of actors in their entirety along with audio. Their third stage uses 40 MX F40 cameras and is customized for particular projects.

"Our previs technology allows multiple actors (up to 15), props and set pieces along with tracked cameras to be captured in realtime to drive multiple associated CG characters within any 3D environment," adds Roberts. "We can provide good real time rendering in 3D and 2D, with lighting, particle, textures and shaders. We can also overlay CG characters with video back plates and camera tracks whenever needed. All this can be used on site here at HOM and on location for any shoot whether it's for on set live-action integration, greenscreen or a more traditional motion capture shoot."

Previs is also used to provide immediate playback on set of any previous shot captured to allow directors and animators to review shots on the fly as they shoot and to make on set selections thus speeding up the post production phases. "We also use pre-visualization on set during camera tracking," Roberts points out. "With this technique we capture the position, orientation and lens of multiple virtual cameras on set as they move, while simultaneously rendering the realtime view from each of them into the virtual world; back to the eye piece or viewfinder of the practical cameras on set. This allows camerapersons, DPs and ultimately the director to take shots in the virtual world, to frame and compose each shot. It even can allow them to shoot their DLO (Directors Layout) while on set. Similarly, virtual traditional camera controls (Wheels) can be used to allow the manipulation of virtual cameras in the 3D world on set as we capture performers or as an on set post tool just after capturing the performers.

"The use of camera tracking on set and as a post tool on set is incredibly powerful for directors and our clients. It allows full creative control of framing, composition and direction on set using more traditional filmmaking techniques. It's amazing to see a talented director and DP work together with actors and these tools on set."

HOM uses realtime previs for many different objectives: on set characterization, previs for animatics, scene and set layout and blocking, painting sets, motion layering, camera tracking, director's layout, dailies, previs for performance capture selection process, compositing, greenscreen, live-action integration and live principal camera tracking. They also use it for final render to air and rest pose blending reference, which some may not find too familiar.

"Often our realtime previs render and animation has been used for live to air or live to broadcast features for various TV stations around the world. Even live to tape. Obviously this brings a whole new level of stress to the stage, but the quality of the real time previs technology here based on MoCap is really that good. Clients often feel comfortable to use it for live to air recordings.

"Rest Pose Blending reference is something our game clients use a lot. Often for videogames, motion trees are created to enable game developers to categorize types of motion for use in the game. These categories need to be transitioned into and out of for fluid animation during the game depending on the players inputs and AI. Many games use the concept of known poses, often called rest poses or blend poses, to transition into and out of. These poses are uniquely identified on set using our pre-visualization tools. They are also references during all game captures or shots to ensure the actor approximately started, transitioned and ended in particular blend poses. This greatly reduces the amount of animation work done in postproduction once the shoot has completed. Here at HOM we have an animation team that provides full animation services to clients and we use the on set previs tools every day for game clients that require these services and even if they want to do the animation work themselves.

"We recently shot a Miller Lite commercial (Break Away From the Crowd) with Zack Snyder directing. Believe Media, Halon and Method Studios were all involved and present onset during the realtime motion capture sequences. Halon provided realtime previs onset with the motion capture data to depict the Miller Lite (60') Giant, made up of humans, walking around downtown L.A. to the beach. Working closely with us, Halon integrated and displayed their character in realtime, overlaid and composited with backplates to enable Zack to direct and get the shots he wanted. We were also able to play the actors' performance back in slow motion to get the weight and feel that was needed. Both director and actor used the realtime MoCap previs and Halon's realtime previs to get the feeling and weight behind the character."

In addition, Target's Product People spots were shot with Radium VFX, all using realtime previs on set at HOM. "Since all of the characters were entirely comprised of home and sports products from the Target stores, previs was essential for the director and actors to modify their performances to bring the characters to life," Roberts continues.

HOM was also used to track the principle camera for the Muse music video Invincible. "The entire music video has the band on a virtual theme park ride, which was all shot on a greenscreen stage in Hollywood," Roberts explains. "HOM provided camera tracking and realtime previs for the virtual CG environments to be composited in realtime with the live-action back plates in realtime to give the DP, director and band members reference while on the virtual theme park ride."

Recently HOM completed shooting several episodes of Sports Science and Fight Science, which will be airing on FSN this year. "Realtime previs was used on set to aid in scientifically analyzing and visualizing the remarkable feats performed by all the top athletes, sports men and fighters in the world," Roberts offers.

What's in the future for Vicon and HOM? "We are continually advancing previs tools that are based and integrated with our motion capture systems," suggests Roberts. "RT previs on and off set for all aspects of traditional filmmaking is something that is very important to us. Our new realtime principal camera tracker will be coming to bear on current productions very soon."

The Pixel Liberation Front (PLF) is also using the Vicon system. "We own a Vicon MX13 FK Extreme motion capture system," says PLF's Exec Producer Sean Cushing. "We are currently utilizing eight cameras, but are considering adding more cameras to accommodate larger scale jobs. In general, the system is typically set up inside our facility but is definitely portable for any on-set MoCap needs. In order to capture some dynamic fight sequences for a project we are currently working on, we have moved the system off-site to take advantage of a wider capture volume."

Recently PLF collaborated with Giant Studios on a MoCap pipeline that was utilized on a large-scale feature film. "The pipeline was unique in that the MoCap data was captured at a resolution that was functional for finish animation work but was also simultaneously processed for us to use immediately in our previs MoCap pipeline." So, what does this mean in terms of efficiency and cost savings to be able to upscale their previs sequence?

"Previs sequences are more and more looked to as tools to work out emotional content and overall storytelling. Therefore, the performance of the previs characters has had to improve dramatically over the years. However, better character animation takes time. Previs has to be fast and interactive. The quality and emotional impact are at odds with the process and the needs of the production schedule.

"To answer these issues, PLF has tapped motion capture technology. We break action down into discrete movements and capture them separately, to be blended together later in the context of the shots. We capture alternate motions as much as possible to allow for fast changes in the sequence without having to capture new motion. We often capture secondary characters simply to get them moving so that the shots seem more realistic. Compared to keyframed character animation done on the same time scale, the quality is orders of magnitude more refined. In addition, when such realistic motion is on the screen, it's quickly accepted and taken for granted, allowing the filmmakers and previs artists to focus on the filmmaking rather than the animation. This is, of course, is the intent, and it appears to be working out quite well."

What does Cushing see for the future of previs MoCap? "Our goal is to develop a pipeline where our previs MoCap data will be of high enough resolution and contextual relevance that it will be useful all the way through to the final shot delivery."

Traditional marker-based motion capture is used to create realistic performances for skeletal captures but what about the intricacies of soft tissue motion such as pursing lips or billowing fabric? The problem has apparently been solved through collaboration between Vicon and San Francisco-based Mova, the developer of the Contour Reality Capture System. Mova's markerless capture system was designed to be used simultaneously with Vicon's MX series marker-based capture system. "Mova's Contour takes us from motion capture to reality capture," explains Mova President Steve Perlman. What this mean is that it is now possible for Contour's cameras to capture high-resolution surface motion, such as facial, skin and cloth motion, while the MX40 cameras capture high-precision marker motion, such as skeletal and prop motion.

But Contour doesn't just capture a scene in 3D; it tracks the polygons in a scene from frame-to-frame. The reason this is critical when it comes to workflow is that it allows faces, characters or cloth to be retargeted. So a face, for example, that is captured can be altered to appear younger or older, or even transformed into an entirely different face in realtime and still retain the nuanced performance of the actor. In the case of a videogame, a performance could be used to drive a lower resolution model, and the videogame developers could decide what facial features are most distinctive for conveying a character that might be played by a celebrity.

As for on set previs, Perlman states, "Since Contour is a photographic system, the reconstructed 3D images captured are generated automatically. During the shoot, the Contour system provides the director, vfx team and actors with reconstructed 3D captures to evaluate on the set. These are not skeletons or simple cartoon characters, which is all that is typically available for realtime marker-based on set playback. These are detailed 3D reconstructions of the actual actor's face and/or body and clothing, accurate to sub-millimeter precision. It looks just like the actor as if he or she were shot with HD video. The difference is that it is completely in 3D, captured volumetrically in gigavoxels rather than megapixels, so you can move around the image and view it from any angle. Thus, Contour delivers data that is effectively impossible to recreate cost-effectively using any other method."

Although Contour received a glowing endorsement last year from David Fincher, Perlman is not able to confirm its involvement in the director's reverse aging fantasy with Brad Pitt, The Curious Case of Benjamin Button. "We are working with A-list actors and directors, on major motion pictures and on breakthrough videogames. Also, I can say that everyone we've done a test with is continuing to work with us. And, I can say that what we are hearing from the people using the technology is that it is revealing subtleties in facial motion that they have never seen before. And, yes, an A-list actor asked us to do a private capture of his face to archive a perfect 3D record of himself at his current age. But we can't tell you who, what projects, etc.

"As an example, Contour is really transforming the way people are using FACS (Facial Action Coding System). Before Contour, the actor was holding his/her face rigidly in each FACS pose as the face was scanned. Well, it takes a lot more muscle strength and different flexing to hold a face in an expression pose than it takes when the face naturally passes through an expression pose. So, the face is shaped differently that it is during a normal performance. With Contour, FACS poses can be captured during a real performance while the face is in natural motion. And, the results are far more natural looking, and don't look as stiff or strained."

According to Brian Nilles, CEO of Vicon U.S., "The innovation by the team at Mova is quickly coalescing into the most profound improvement in CG facial animation since our own MX40 camera. For CG animation that requires both the performer and the CG character to be the same, the Mova Contour system should provide a clear and significant step toward the holy grail of a photorealistic final product. We are proud to be both collaborator and integrator for Mova's work with the Vicon system."

J. Paul Peszko is a freelance writer and screenwriter living in Los Angeles. He writes various features and reviews as well as short fiction. He has a feature comedy in development and has just completed his second novel. When he isn't writing, he teaches communications courses.