Breaking the Fourth Wall with John Gaeta

Since working with the Wachowskis on The Matrix movies and Speed Racer, John Gaeta has been experimenting with motion sensing technology (Kinect) as a way of converging movies and interactive media into a more compelling, mind-bending and unifying experience: In other words, breaking the fourth wall. We had a wild conversation, which was streamed live on Sunday at the inaugural Palo Alto International Film Festival.

Gaeta began by tracing Eadweard Muybridge's pioneering experiments in photographic motion (appropriately in Palo Alto and the festival's iconic logo) to his "Bullet Time" innovation in The Matrix to the current Kinect work with games that he's doing at his Float company in San Francisco.

Then Gaeta offered a provocative scenario: "The whole sensor revolution, really, is starting to pour itself in all manner of application," he suggested. "For instance, the only way to port people in a holographic way would be real time spatial acquisition of them and their textures and to bring them into some common viewing space… Over time, people have gotten better and better at combining technologies like HD video and image capture and other types of spatial acquisition to come up with a way of making a virtual construct of it.

"You're about to have sensors in your life in a most prolific way, so your life is going to be metricized at some point in the next 10 years, including facial recognition. And by choice, billions of people are going to put sensors in their homes so they can get all sorts of cool shit -- super immersive entertainment. They'll be able to interact with their televisions and their mobile devices and then there won't be TVs anymore and you'll have your big, immersive boom room."

But what will it mean for movies?

It's Gaeta's contention that while the movie industry is creatively stagnating, we're on the verge of a new renaissance of technological innovation that will transform both movies and interactive entertainment into a deeper and more subjective experience. We're talking holographic immersion with complete volumetric capture so viewers can watch from the viewpoint of the director or select their own individual perspectives. In other words, imagine going into the Matrix with Neo, or being transported into the watercolor heaven of What Dreams May Come, let alone the mysterious Pandora or the nightmarish Occurrence at Owl Creek Bridge (one of Gaeta's faves).

Gaeta underscored that there will always be one thread of movies experienced on the big or small screen because of the power of compelling performance sculpted by directors in a way that we can't construct ourselves. "But I do think that the universe these auteurs create is a place I might want to go deeper into," he offered. "So you could [experience] it the way they sculpt it, or by toggling off the camera I could get a free view version and see the whole film from any angle that you want. But it's still the performance: I'm not bending the performance, I'm not changing it, I'm not choosing an alternate angle. And I can look all around that world and get all sorts of incredible, expository information, and deepen my understanding of the characters and the world as a dividend of just having the free view.

"So all I'm saying is that the next major revolution is gonna be omni-capture of the movies and augmentation of all that stuff is not going to be dissimilar, except it'll be done with real time technology, so that, yes, we can record the world around. But there will be a time that comes in the next decade where the film will be made volumetrically and the ability to nav and explore whatever. You can serve it any way you want. You can serve it stereoscopically -- there will be holographic display systems that people will get in the next decade, for sure. There will be other kinds of formats that are more immersive, without doubt. It's a redirective, existing technology."

There will also be the possibility of having blended, augmented reality in the near future. "Just imagine how indelible it would be if I interacted with something, and if something absolutely remarkable and inspirational occurred. I think there's potential for some really good experiences."

The threat of Big Brother aside, the imagination reels with creative possibilities.

Bill Desowitz is former senior editor of AWN and editor of VFXWorld. He has a new blog, Immersed in Movies (www.billdesowitz.com), and is currently writing a book about the evolution of James Bond from Connery to Craig, scheduled for publication next year, which is the 50th anniversary of the franchise.

Bill Desowitz, former editor of VFXWorld, is currently the Crafts Editor of IndieWire.