Programming tool makes bugs sing TRN 080702

Programming tool makes bugs sing

By Kimberly Patch, Technology Research News

Over the past few centuries we've found ways to cram more data into graphics like maps, and dynamic elements like slider bars have allowed computers to display even more data per inch. Sound, however, is largely absent from these tools, despite music's long history as a universal language.

Researchers from the University of Northumbria in England are tapping the auditory sense by allowing programmers to listen, rather than simply look, for software bugs.

The work may eventually give sound a bigger role in computing. It is also a step toward making programming and other computing activities more accessible to people who are visually impaired.

The researchers first tested the ability of the average non-musician to distinguish differences like pitch using sounds similar to those musical instruments make. The results were good. "Most people could discriminate between separate pitches, and describe melodic contours quite well," said Paul Vickers, a principal lecturer at Northumbria University.

They then set up software that mapped pitch and melodic contour information to structural elements in the programming language Pascal. "[We] aimed to see if information about the structure of Pascal programs could be communicated using such musical phrases," said Vickers.

The researchers used similar musical phrases, or motifs, to represent similar programming constructs.

The constructs of the Pascal programming language can be categorized hierarchically into two classes -- selections and iterations -- which, in turn, have sub-classes, Vickers said. For instance, in the iteration class, the language has a pair of similar bounded loops,: "FOR... TO" and "FOR... DOWNTO". It also has a pair of unbounded loops: "REPEAT" and "WHILE".

The researchers used a common theme for each class, and wrote musical motifs that were variations on that theme to make the similar REPEAT and WHILE loops sound distinct, but more similar to each other than to the bounded loops. At the same time, all four of the phrases representing loops sounded similar enough to each other that they could be distinguished as being in the iteration class rather than the selection class.

Once again the results were good. "Programmers [can] derive useful program information from musical representations," said Vickers.

Once they had established the basics, the researchers worked out a melodic system for debugging that follows several basic principles.

In writing the motifs, the researchers used the common diatonic, seven-note scale, which is easy to memorize and recognize, and they made use of several sound variables that are easy to follow.

They used meter and rhythm, which previous music cognition research showed are easily retained, to delineate different parts of the program. A song remains recognizable, for example, even when the underlying cords are changed significantly, if its basic rhythm stays intact, said Vickers. The meter, or time signature of a musical phrase indicates a certain number of beats per measure.

They added percussion to changes in melody. "Percussive devices can enhance the music and provide extra clues... to help users recognize significant events such as a change in condition of a Boolean evaluation," said Vickers.

And they used continuous tones, or drones similar to those used by bagpipes, to indicate continuous states like loops where many nested operations may take place. "The use of a continuous tone can indicate that the program is inside the loop," said Vickers.

Once the researchers worked out the musical rules, the scheme was easy to implement, said Vickers. "We used simple Musical Instrument Digital Interface (MIDI) devices," he said.

When the researchers tested the auralization tool on 22 undergraduate computer science students, they found that the subjects could correctly locate bugs in a program more often when sound was added. What surprised the researchers was the response times for subjects who used the sound and those who did not were similar, said Vickers. "Because the musical auralizations took time to listen to -- up to two minutes in some cases -- we might have expected... an increase in the time spent locating the bugs," he said.

The researchers work is a good exploration of the ways people can absorb information presented musically, said Mike O'Donnell, a professor of computer science at the University of Chicago. "We have so little information on that topic that every experiment is very helpful," he said. "I'm not aware of anyone else using musical phrases to render the execution of a computer program."

There are at least two challenges to using sound to represent information, said O'Donnell.

First, time is perceived in at least three different ways and these three ways have radically different structures: the speed of vibration, or frequency, which we hear as a certain pitch; sudden changes in the frequency, which we perceive as part of the timbre that makes up a particular type of sound; and sequences of sounds over time, which we perceive as, for instance, a sequence of notes in a musical melody.

The problem is that these three different ways of perceiving overlap. "We can't separate them cleanly," said O'Donnell. In contrast, a moving picture can be separated into a rapid sequence of static frames.

Second, because aural information is based on a sequence of sounds over time, its perception must be explicitly programmed. In contrast, we can browse a static picture by shifting the focus of our eyes, said O'Donnell. "Eye motion generates a lot of valuable interaction between a human user and a graphical display for free," he said.

It would be interesting to condense the musical material so that a more experienced programmer can listen for patterns in much larger program runs, said O'Donnell. Looking at the concept backwards may also be enlightening, he said. Writing programs that make interesting musical segments "may give [programmers] an intuition for the impact of different programming constructs," he said.

The researchers are currently looking at how musical representations of programming languages might be used in teaching computer programming, said Vickers.

Another line of research is to see how auditory signals can be used to build accessible user interfaces, he said. They are "investigating using musical messages to assist elderly users with monitoring and controlling household devices," he said.

Sound has several potential benefits for human-computer interaction that could eventually be exploited, said Vickers. "Today's graphical user interfaces are increasingly complex, leading to cluttered screens and a general lack of screen real estate. If some of the communication burden can be transferred onto the audio channel, then perhaps interfaces can be made more efficient, more effective, less cluttered [and] easier to use," he said.

Sound could also help visually impaired users in applications like audio-enhanced Web browsers, he said.

Sound also offers a new paradigm for exploring data, he said. "Its ability to transmit multiple streams in parallel -- consider the different instrumental parts of the symphony -- and its ability to transmit in time-based rather than spatial domains offer us new ways of interacting with data," he said.

Vickers' research colleague was James L. Alty of Loughborough University in England. They are publishing the research in an upcoming issue of the journal Interacting with Computers. The research was initially funded by Liverpool John Moores University.

Timeline: 4-5 years
Funding: University
TRN Categories: Computer Science; Human-Computer Interaction; Multimedia
Story Type: News
Related Elements: Technical papers, "Using Music to Communicate Computing Information," slated for publication in Interacting with Computers; "Musical Program Auralisation: a Structured Approach to Motif Design," Interacting with Computers, slated for publication in Interacting with Computers; "When Bugs Sing," slated for publication in Interacting with Computers.

Advertisements: