Visions of the Future for Computer Graphics and Communication

Talk to NYU Professor Dr. Ken Perlin about the future of computer graphics, more specifically, user interface, interaction and communication, and you come away with the sense that he has mastered the difficult skill of distilling complex scientific reasoning into ideas that are contextually relevant and understandable to a person of average intelligence.  Like me. Over the past 30 years, I’ve spent considerable time talking to scores of really smart people and I’ve watched more speakers deliver bad presentations than toothless Yukon gold miners have watched their sluice boxes wash out empty.  If the future of mankind hinged on a computer conference speaker's ability to explain how to prepare a box of macaroni and cheese, best dust off that 21-year old bottle of Auchentoshen you keep for special occasions and savor it with haste because we’re all going down in flames, right after Professor Clarity gets to the 7-step process for opening the flavor packet. Complete with flowcharts. With Ken, however, it was different.  He explained stuff, I actually understood. No flash cards or lifelines. How refreshing. 

In 1997, Ken won an Academy Award for the development of Perlin Noise, a technique used to produce natural appearing textures on computer generated surfaces for motion picture visual effects. In 1997, I built a FoxPro database that crashed my Mac so badly it had to be reformatted. We both studied math in college, but evidence suggests he understood it a bit better than I did.

Ken’s understanding of history, the societal impact of innovation and discovery and how the context of people’s understanding and acceptance of technological change is key in assessing possibilities for the future of computer graphics, is central to how he explains his vision for that future.

I got a chance to spend some time with him this past December at SIGGRAPH Asia in Hong Kong, where he was presenting a talk on the future of computer graphics.  I’ve never thought of a piano as a communication technology, because, well, it’s just a piano and there’s no power cord and my wife can’t yell at me through it.  But after speaking with Ken, I’ve started looking at everything in my surroundings, including my awesome new potato ricer, in an entirely new light. Considering that no GPUs are involved, nothing communicates happiness better than really fluffy mashed potatoes.

Dan Sarto: What will you be talking about in your presentation this week.

Ken Perlin: Mostly I’ll be talking about my interest in using computer graphics as a means of getting people to communicate directly with each other.

DS: What do you see as some of the future trends in the use of technology and interfaces for human interaction and communication?

KP: In all technologies, the success of that technology is measured by the extent to which we forget it exists.  You do not “access” your automobile.  Instead, you go across town. The fact that you can just say, “I went to the store and bought something” means that the automobile is a successful technology because “automobile” never appeared in that sentence. And actually, before the smartphone, the telephone was a very successful technology because no one ever “accessed” a phone; they just talked to their friend in California. A very simple, very mature interface. You don’t notice refrigerators or air conditioners. Technologies that really work fade into the woodwork. I think computer graphics is still largely at the stage where it’s about “computer graphics.” I went to this movie and I saw these special effects. All this cool stuff on the screen, or on my iPad.   Ultimately, we would like, in the long run, for computer graphics to merely be an inseparable part of how we communicate with each other, and that communication will be better.

DS: Do you see technology pushing the dynamic of human interactions, or do you see technology filling voids and just further enabling what people are already doing by themselves naturally?

KP: All technologies that are communication technologies, whether they are the pen, or the piano, or the white board, or paper, they all do this dance. It’s not that the arrow goes one way, it’s that the arrow continually goes back and forth, back and forth. There are always many, many technological innovations that are happening at any given moment. Those that turn out to be useful to people are ones that get pushed on more. So it’s a continual process of mapping possible technological advancements to the way it turns out people really work when that technology is presented to them. For example, in the late 90s, there wasn’t any a priori way of knowing whether the expert driven search offered by Yahoo was going to be the right way to do things. It turned out that the crowd sourced search offered by Google was a better match for what happens when people need to do search on a massive scale. When I say crowd sourced I mean that the search results start fitting themselves to people’s actual patterns of use, as opposed to relying upon a team of experts to figure that out. 

DS: Where do you see the balance between pure academic research and market driven research based solely on perceived commercial need or opportunity?

KP: Many people [researchers] find it exciting to work on things that people will actually use. There is an asymmetric relationship. By analogy, the successful playwright/ author and the audience that sees these plays or reads these novels, each is essential. The audience doesn’t know how to make what it wants, but it’s an incredibly good filter for whether people are making what it wants or don’t make what it wants. In any consumer - producer relationship, both are drivers. So, I do think that people who work on innovations deep down want to work on things that are going to be relevant to people who might use them. I don’t see that there is that much of a conflict in terms of what everybody wants. 

DS: Where do you see the future heading for areas of user interface research, which is an area that is influenced so heavily by huge consumer opportunities and market forces?

KP: It’s not as simple as it might first appear. All of these things [practical applications of research] require staging. There are pre-conditions. For example, if someone had said, soon after Dennis Gabor and others had invented the laser back in the late 50s, which was being presented as the death ray of the future, that one day, millions of people will deliberately scar their corneas as a way of seeing better, people would have thought that person was crazy, or maybe on drugs. But now, Lasik surgery is taken for granted. The reason is that things happened in stages. There was something, then there was something else, then something else, and then expectations changed, the technology advanced, the sociology around it changed. I think you find that with lots of technologies. 

There was strong belief a little over a century ago that it may not be possible for human beings to go beyond 60 miles an hour and live. It took a number of decades before this idea [traveling faster than 60 miles per hour] became a normal part of existence. 

One of the key things to remember about augmented reality technologies is that they’re going to require a series of changes even in display which have to happen in stages, they can’t happen all at once. One thing to keep in mind is that we now think of the iPad and things like the iPad, these tablets, as something very natural in some way.  In 1968, Alan Kay came up with this [tablet] form factor. But there was so much that didn’t exist.  For one thing, the widely crowd sourced Internet didn’t exist until 25 years after that.  From 1968 to 1993, that’s only 18 years ago. And a whole series of evolutionary steps had to happen after that. 

So we can talk about possible futures, but we can’t predict them, because there are too many different staged steps that are not just technological but are sociological reactions.  Okay, people are ready for Lasik now. People are ready to get behind the wheel of a car. Microsoft did a very noble thing in 2001 by trying to make the tablet PC.  But the whole ecosystem wasn’t ready yet. There were fundamental pieces missing from that market play. Apple took a big step back, watched that happen and did a whole series of staged efforts over the course of a number of years. The iPad is part of a whole series of things they did that started much earlier.

DS: What do you see as some of the next stages or phases with regards to the merging of graphics, user interface and technology leading to augmented reality?

KP: One of the important things that I think is only going to be a transitional technology, but a necessary transitional technology, is that sometime in the next 5 years, those glasses that you are wearing will at first, in a very expensive way, and then in a less and less expensive way, be able to be a see-through display. That’s not where things are going to end up. But that will give people this idea of what it is they can have if someone gave them what they wanted. In other words, people have to see it first, and then they start creating all of these uses.

So when Tim Berners-Lee came up with this way of people crowd sourcing internet links,  that wasn't yet Mark Zuckerberg finally figuring out a very successful way to do social networking where people before had not done that, that was a pre-condition. So I see the see through, not bulky display as a pre-condition for people saying, “Oh, okay. Now let me think what I want to do with this. How does this change my everyday life?” And the technologies can start getting refined in the same way that people saw the Silicon Graphics [workstation] and it cost fifty thousand dollars. There wasn't a significant market for a fifty thousand dollar box to do 3D graphics, but that just enabled people to start thinking, “What do we want?”

And then, of course, you've got NDIVIA and its competitors doing the same thing for a few hundred dollars. Once you got to that commodity level, everything took off because now you could pay for all the real R&D at commodity volume. Then per unit things got really cheap.

So we're going to go through a transition. We're going to have the Silicon Graphics version of things. It's not going to be available for everybody, but then enough innovation will happen that people will understand why they want it. I think that one of the most important things that happened, maybe the most important thing that's happened in terms of people fundamentally rethinking communication is not the iPad, it's the Microsoft Kinect. Suddenly, you have a very large number of really smart people, a lot of them in their early 20s, who for the first time have their hands on a technology that can let them track human body movement however they want. They can use the official Microsoft SDK, but they don't have to. And this is changing a lot! Those devices, something like that was available for ten thousand dollars but that's not interesting. As soon as it goes down enough and it's good enough that you get some hacking, so if there were [see through] glasses for a few hundred dollars, that would inevitably lead to the disposable ones. It'll take a while, but once you get to the point where enough people are innovating, the market drives it really far really fast.

DS: You just highlighted an important issue. Who bears the cost of making these groundbreaking steps that move the technology forward to the point that makes it interesting enough for people to pay attention? That's a tremendous burden.

KP: I think that what Apple is showing with the whole iTunes/Apps/iPad/iPhone ecosystem and with what Microsoft is showing with the whole Xbox 360/Halo/Kinect ecosystem is that the people who are successful at taking this risk are the ones who are good at making it a content play. How to connect it to content that people want.  You present the hardware as a better delivery mechanism, an evolution of something that people already understand or can easily be made to understand because it connects with something that they want. You don't just throw hardware into the world and it all just works, it has to be part of a well thought out set of interventions.

DS: Do you think that there are others? You use Apple and Microsoft a lot as examples. As innovative as Apple is, for example, there are a lot of people that are taking them to task with the way that they are integrating rights management with the content, with all of this innovative hardware.

KP: I don't think that anybody outside of Apple is happy about that. And then another complicated case, of course, is Amazon. Amazon is selling all their eBook readers at a pretty significant loss! They lose a hefty amount of money on each one that they sell you, whether it's the Kindle, or the new Fire. With the Kindle classic they're probably losing $25 on each one of those things. And the reason is that they're not trying to make money on those devices, they're trying to make money because they're going to sell you a whole bunch of eBooks and other things through them. So there is a hardware innovator that is throwing away an incredible amount of money to get you into their rights world, their content world.

Do I think it's a Devil's bargain? Of course I do! At some point will people rebel? The danger for those early innovators who then try to lock into content is that eventually somebody will come along who won't lock into content, because now it's a world where that's not necessary. So you can do that for a while, but you can't do that forever.

DS: That comes back to one of the first things I was asking you with regards to the commercial research and innovation.  You make this excellent point about how moving forward, there will be staged growth. Yet with so many of these innovators, it's driven by the financial realities of the commercial application. In the work that you do and you see others doing, do you feel that those economic forces are overpowering the research, or do you still see a lot of research that's being done more altruistically, or with less of an overt commercial aspect?

KP: Every corporation thinks that it is forever. IBM thought that it was forever, Apple is probably under the illusion that it is. Google thinks that it is, but they're not. They come and they go, and they're good at something and then somebody else comes along with something they completely did not see coming, and then they go the way of DEC. So corporations have their time, and in their time they make vast amounts of money and then they're gone.

I think people who are in academic research say it's great that there are these engines of commerce that throw some things out into vast capitalization states, but the work that I'm doing, I don't care what company ends up making money off it, whether it's 5 years from now, 10 years, 25 years from now. The company, in the long run, is just the vehicle for the change that happens in the way people do things. And then the company goes out of business and people still benefit from the set of evolutions. Whoever was predominant in making paperback books in the late 1800s and revolutionizing how people read, that company is not around anymore. But it doesn't matter, the change happened.

So if we really think long term, we shouldn't focus on Apple or whoever the equivalent of Apple is going to be in 20 years.

DS: Fair enough. Tell me a little bit about some of your own research. You have such an excellent perspective on some of the broader issues, but you have your own projects. Tell us a little bit about some of what you're doing and where you see that going.

KP: I've come to think that there is a gap in some of the research and academic focus. C.P. Snow talked about the two cultures. The culture of the arts and the culture of the sciences and technology, and how they have very, very different ways of thinking about things. One of the things that I have loved about the SIGGRAPH community, and the reason that I am in graphics as much as anything else, is that it's the closest we have to those two cultures coming together. It's an absolutely top ACM conference for computer science PhD students to make their chops by publishing papers at an A-rated conference, and at the same time it's a vehicle by which people can express themselves artistically in ways that no one's ever seen before. But there's been a disconnect. I think what's happened is that some of the early pioneers, Myron Krueger, some of the early stuff that Alan Kay was doing, some of the early work that Hiroshi Ishii was doing, a number of other people had been looking at things primarily in terms of, “I'm a person, you're a person, how do I communicate with you? Right now, how do we talk? How do we use technology to make this better?”

And human-computer interfaces, the whole SIGCHI community had veered away from that into, “How do I make a better version of Google SketchUp? How do I make a tool to make something?” As opposed to a tool for me to communicate with you. Every time people try to make a tool for me to communicate with you, it gets a little too computer-sciency, a little too technical, too engineering for my tastes. Too much about “Let's use these engineering assessment measures to see if this works.”

Meanwhile, I think Hollywood has been presenting these visions about what it would be like if we really could communicate better, but they're not really doing it. Cameron gives you Avatar, and then you see the heads-up display that Iron Man has, and then you see whatever is that crazy interface that alien had in District 9. So there are these visions of what we really want, for the technology to serve us and communicate in this very seamless way.

So I'm interested in seeing these cultures of the arts-infused visions of “What do we want,” connected with the kind of engineering expertise to make it happen, and to pull together something that's not about trying to get this task done, but rather, “I'm trying to use this stuff to communicate with you,” which is not pure technology. It can't be pure technology, it has to be a lot of people with great artistic insights, great intuition, hacking on things until we figure out what really works between people. That's not going to be done by human factor studies.  That's going to be done by a wide ranging playful exploration of the possibilities. I've become really interested in trying to find that thing that's between SIGGRAPH and SIGCHI that's about interactive human-to-human communication.

DS: Last thing.  From a purely personal standpoint, at this point in your career, what gives you the most sense of satisfaction?

KP: When people get together and do things with technology and we all feel this genuine sense of wonder at the possibilities of how to better communicate with each other, that's when I feel it. I'm working on that and other people are working on that as well, so I see that sense of wonder in a number of places. I certainly see it in our own work, but certainly not only in our own work. There is a community of people who are interested in that.

--

Dr. Perlin directs the NYU Games For Learning Institute. He was also founding director of the Media Research Laboratory and director of the NYU Center for Advanced Technology. For more information about his work, visit his website at http://cs.nyu.edu/~perlin/. 

Dan Sarto is publisher of AnimationWorld Network.