Karen Raugust investigates a Purdue University research project using 3D animation and virtual reality to teach math concepts to hearing impaired school children.
Two Purdue University professors are studying how to use computer animation and virtual reality to teach deaf children in grades K through 4 about mathematics principles, and to communicate math terminology using American Sign Language. The reason for the project is that, for deaf children, reading instruction is delayed; therefore most kids in this age group cannot use traditional textbooks. They also arent able to take full advantage of secondary learning opportunities that non-disabled kids have, such as television and dinner table conversation, and their parents may not be fluent enough in sign language to teach them about math.
The Purdue project hopes to alleviate these challenges by allowing deaf children to learn math concepts from animated characters, who use sign language to communicate with the kids in a computer-game setting. Even more effective is extending the program into virtual reality, where the characters and the children can communicate with each other in sign language in real time, in a fully immersive environment.
Three years ago, Nicoletta Adamo-Villani, assistant professor in Purdues Department of Computer Graphics Technology, who teaches 2D and 3D animation and specializes in character animation and design, joined with Ronnie Wilbur, professor and chair of linguistics in Purdues Department of Speech, Language and Hearing Science. The first phase of their joint project focused on creating an interactive PC game called Mathsigner, in which 3D avatars communicate with the user in sign language. The team recently earned a grant from the National Science Foundation to proceed with this aspect of their work; it is currently in prototype form and in use at the Indianapolis School for the Deaf as a web/CD-deliverable desktop application.
A more recent phase of the project extends the software into an immersive, virtual-reality world. Working with Purdues Envision Center for Data Perceptualization, the researchers are creating a game that is projected onto three walls and the floor of a special room, using Fakespace Labs FLEX system. (The VR game can be played on a PC as well but it is not immersive in that format.) When using the application, the child wears a special headset, lightweight stereoscopic glasses, pinch gloves and a wrist tracker. These devices allow the child to communicate with the avatars and redraws the animation to follow the childs perspective as he or she moves.
Research shows that immersive games are more effective than non-immersive games, says Adamo-Villani. Studies have found that humans process visual information 60,000 times faster than text-based information, she reports, noting that students can raise their math scores 16% after eight weeks using an immersive VR program.
Such a system is an effective teaching tool because it gives the students control over the environment, offers self-paced and repetitive learning, presents concepts in a concrete rather than abstract manner, features an environment that feels safe, and, as a game, is motivational. All of these characteristics help the student overcome challenges that are common to people with learning disabilities.
The Animation Process
The game features four stores in which students learn a different math concept: a bakery where they learn about weight, a clock store where they learn to tell time, a toy store where they learn how to count money and a candy store where they learn addition and subtraction. The avatar and the child can communicate; for example, one of the avatars can ask a question and the child can sign a simple answer, such as a number, or the child can pick up a certain number of candies and the avatar can make the sign for that number.
Motion-capture is used to record the signs, so the avatars characters such as rabbits, robots and pigs that serve as the storekeepers can communicate with the students in real time; the remaining animation, including facial expressions, is key-frame. All models were created in Maya.
In order for the Mathsigner characters to be displayed in the FLEX system, the polygon count had to be reduced to maintain the fast frame rate and real-time interaction, and new facial rigs had to be created. Polygons are concentrated in areas where detail is needed most, namely parts of the body that bend or twist. This keeps the signing realistic-looking.
It requires an extremely fluid motion, says Adamo-Villani. Were working with signers who are very picky. The motion has to be very realistic. At the same time, the animation look is colorful, cartoony and stylized. Were using Disneys Toontown as inspiration, she explains, noting that while many of the challenges of this project are similar to any animation production, a unique challenge is to make the stylized characters do the signing motions organically, in real time.
The project is popular with students in computer animation at Purdue. They see it as a good opportunity to learn both motion capture and key-frame animation, and they end up with a strong portfolio piece. Graduate, masters and undergraduate students are working on it, led by David Jones, a graduate student in the department of computer graphics technology.
The application is being refined to add more gaming elements. Currently, users can go into individual stores, one at a time. The creators are looking at ways to link the stores so the child can move from one to another as he or she completes a task, making the action more like a videogame.
Feedback from children led the animators to add more storytelling as well. When the kids were testing it, we got comments like, Why dont you have a story? says Adamo-Villani, who previously worked on the Ironman television series at Marvel, as well as in computer graphics research at the University of California-Riverside.
Some improvements are under consideration on the technology side, too. The researchers plan to look into using more sensitive gloves, such as those employed for mo-cap, which would enable more complex signing by the children as they use the immersive system. In addition, the current FLEX system is expensive and must be used in a special room, which makes it impractical for both institutions and individuals. The researchers are exploring different options for a portable yet still immersive system. Thats one of the challenges of the project, Adamo-Villano says. The technologys not there yet.
For more information visit Mathsigner at www2.tech.purdue.edu/cgt/I3/ and SMILE at www2.tech.purdue.edu/cgt/I3/SMILE/.
Karen Raugust is a Minneapolis-based freelance business writer specializing in animation, publishing, licensing and art. She is the author of The Licensing Business Handbook (EPM Communications).
