Fun with Sound

Musical compositions and airport security have more in common than you might think.

Musical compositions and airport security have more in common than you might think.

The link is a novel combination of a black room and web cameras that enables anyone – whether engineers or artists -- to tinker with the interplay of sound and movement. The Interactive Studio is part of an overall effort which includes the Duke Immersive Virtual Environment (DIVE) that probes how technology can be used to visualize data.

In the larger sense, the goal is to better understand how computers understand and communicate with humans, and more specifically, to enable computers to create images based on sound. On another level, the facility also serves as an interactive venue to stimulate students into exploring science, technology and engineering.

All the action takes place in a room in the Fitzpatrick Center where the floor, walls and ceiling are painted black. Criss-crossing the floors are colored strips of masking tape. Computer monitors hang on the walls.

“We installed web cameras in the ceiling and aimed them at distinct areas of the room,” explains Steve Feller, Duke engineer and one of the facility’s lead designers. “Each camera streams images to a computer that figures out how much motion is happening in the scene for each camera. The computer processes the images, sends them to sound-rendering computers, which then generates the sounds.”

The specific location of motion determines which unique sounds are created. Depending on which sound program is in use, the sounds can include such familiar outdoor noises as wind chimes, rain, lightning; musical instruments including the marimba, gongs, voices, synthesizers, and a variety of percussion instruments.

The amount of motion in each area changes the intensity and volume of the experience. In some cases, spinning beneath a camera sends sounds swirling around the room or causes thunder to boom dramatically.

The ability to transform movement into recognizable sounds presents intriguing security possibilities. Rachael Brady, director of Pratt’s Visualization Technology Group, uses the example of monitoring at a busy airport, where hundreds of people are hustling and bustling about. Typically, security personnel must keep their eyes glued to a video monitor.

She believes that sound may be better than vision at detecting suspicious activity and sounding the alarm. Instead of looking for aberrant movement, security could listen for tell-tale sounds.

“If there is high activity – such as a person darting against the flow in a certain space – the video camera would sense that movement and turn it into a sound that someone would understand as suspicious,” she said. “This would be a way of conveying the state of a space to a human through the auditory channel instead of through the visual channel, which is what usually happens with closed circuit TV.”

While some may see the space as an ominous and cold black room, Scott Lindroth, who teaches music and composition at Duke and is part of the team, often looks at the room as an actual musical instrument.

“I think of the room as a giant musical instrument that somebody could learn to play in such a way that their movements could constitute a musical performance,” Lindroth said. “It would be exciting to collaborate with a choreographer to design movements that play the instruments in a precise way.”

The scientists believe that the facility can also serve as an accessible entry for students to get interested in science. This could be true for youngsters who initially see the room as a cool place to play around in, or college students learning the latest in image processing, parallel computing, network interfaces, visualization, and sonification.

In fact, this summer the team duplicated the facility at the Museum of Life and Science in Durham. So far, more than 70,000 visitors have “played” the facility since its opening June 21.

“This is a technology that I believe is accessible to everybody and will make them interested in science, math, engineering and technology,” Feller said. “Our theory is that if we can get them to understand how this room works and realize that this really is college level research, then hopefully the experience will spark their interest in mathematics and science. If we can manage to do that during middle and high school, hopefully it will stay with them for life.”

One of the plans is to develop a website where anyone can go and download the software and create their own mapping of sounds. People could then share their sound files with other people through the internet.

“In that sense it’s a very interesting social mechanism to explore multicultural aspects of the internet and of music and of science,” Feller said.