tool makes bugs sing
Technology Research News
Over the past few centuries we've found
ways to cram more data into graphics like maps, and dynamic elements like
slider bars have allowed computers to display even more data per inch.
Sound, however, is largely absent from these tools, despite music's long
history as a universal language.
Researchers from the University of Northumbria in England are tapping
the auditory sense by allowing programmers to listen, rather than simply
look, for software bugs.
The work may eventually give sound a bigger role in computing. It is also
a step toward making programming and other computing activities more accessible
to people who are visually impaired.
The researchers first tested the ability of the average non-musician to
distinguish differences like pitch using sounds similar to those musical
instruments make. The results were good. "Most people could discriminate
between separate pitches, and describe melodic contours quite well," said
Paul Vickers, a principal lecturer at Northumbria University.
They then set up software that mapped pitch and melodic contour information
to structural elements in the programming language Pascal. "[We] aimed
to see if information about the structure of Pascal programs could be
communicated using such musical phrases," said Vickers.
The researchers used similar musical phrases, or motifs, to represent
similar programming constructs.
The constructs of the Pascal programming
language can be categorized hierarchically into two classes -- selections
and iterations -- which, in turn, have sub-classes, Vickers said. For
instance, in the iteration class, the language has a pair of similar bounded
loops,: "FOR... TO" and "FOR... DOWNTO". It also has a pair of unbounded
loops: "REPEAT" and "WHILE".
The researchers used a common theme for each class, and wrote musical
motifs that were variations on that theme to make the similar REPEAT and
WHILE loops sound distinct, but more similar to each other than to the
bounded loops. At the same time, all four of the phrases representing
loops sounded similar enough to each other that they could be distinguished
as being in the iteration class rather than the selection class.
Once again the results were good. "Programmers [can] derive useful program
information from musical representations," said Vickers.
Once they had established the basics, the researchers worked out a melodic
system for debugging that follows several basic principles.
In writing the motifs, the researchers used the common diatonic, seven-note
scale, which is easy to memorize and recognize, and they made use of several
sound variables that are easy to follow.
They used meter and rhythm, which previous music cognition research showed
are easily retained, to delineate different parts of the program. A song
remains recognizable, for example, even when the underlying cords are
changed significantly, if its basic rhythm stays intact, said Vickers.
The meter, or time signature of a musical phrase indicates a certain number
of beats per measure.
They added percussion to changes in melody. "Percussive devices can enhance
the music and provide extra clues... to help users recognize significant
events such as a change in condition of a Boolean evaluation," said Vickers.
And they used continuous tones, or drones similar to those used by bagpipes,
to indicate continuous states like loops where many nested operations
may take place. "The use of a continuous tone can indicate that the program
is inside the loop," said Vickers.
Once the researchers worked out the musical rules, the scheme was easy
to implement, said Vickers. "We used simple Musical Instrument Digital
Interface (MIDI) devices," he said.
When the researchers tested the auralization tool on 22 undergraduate
computer science students, they found that the subjects could correctly
locate bugs in a program more often when sound was added. What surprised
the researchers was the response times for subjects who used the sound
and those who did not were similar, said Vickers. "Because the musical
auralizations took time to listen to -- up to two minutes in some cases
-- we might have expected... an increase in the time spent locating the
bugs," he said.
The researchers work is a good exploration of the ways people can absorb
information presented musically, said Mike O'Donnell, a professor of computer
science at the University of Chicago. "We have so little information on
that topic that every experiment is very helpful," he said. "I'm not aware
of anyone else using musical phrases to render the execution of a computer
There are at least two challenges to using sound to represent information,
First, time is perceived in at least three different ways and these three
ways have radically different structures: the speed of vibration, or frequency,
which we hear as a certain pitch; sudden changes in the frequency, which
we perceive as part of the timbre that makes up a particular type of sound;
and sequences of sounds over time, which we perceive as, for instance,
a sequence of notes in a musical melody.
The problem is that these three different ways of perceiving overlap.
"We can't separate them cleanly," said O'Donnell. In contrast, a moving
picture can be separated into a rapid sequence of static frames.
Second, because aural information is based on a sequence of sounds over
time, its perception must be explicitly programmed. In contrast, we can
browse a static picture by shifting the focus of our eyes, said O'Donnell.
"Eye motion generates a lot of valuable interaction between a human user
and a graphical display for free," he said.
It would be interesting to condense the musical material so that a more
experienced programmer can listen for patterns in much larger program
runs, said O'Donnell. Looking at the concept backwards may also be enlightening,
he said. Writing programs that make interesting musical segments "may
give [programmers] an intuition for the impact of different programming
constructs," he said.
The researchers are currently looking at how musical representations of
programming languages might be used in teaching computer programming,
Another line of research is to see how auditory signals can be used to
build accessible user interfaces, he said. They are "investigating using
musical messages to assist elderly users with monitoring and controlling
household devices," he said.
Sound has several potential benefits for human-computer interaction that
could eventually be exploited, said Vickers. "Today's graphical user interfaces
are increasingly complex, leading to cluttered screens and a general lack
of screen real estate. If some of the communication burden can be transferred
onto the audio channel, then perhaps interfaces can be made more efficient,
more effective, less cluttered [and] easier to use," he said.
Sound could also help visually impaired users in applications like audio-enhanced
Web browsers, he said.
Sound also offers a new paradigm for exploring data, he said. "Its ability
to transmit multiple streams in parallel -- consider the different instrumental
parts of the symphony -- and its ability to transmit in time-based rather
than spatial domains offer us new ways of interacting with data," he said.
Vickers' research colleague was James L. Alty of Loughborough University
in England. They are publishing the research in an upcoming issue of the
journal Interacting with Computers. The research was initially funded
by Liverpool John Moores University.
Timeline: 4-5 years
TRN Categories: Computer Science; Human-Computer Interaction;
Story Type: News
Related Elements: Technical papers, "Using Music to Communicate
Computing Information," slated for publication in Interacting with Computers;
"Musical Program Auralisation: a Structured Approach to Motif Design,"
Interacting with Computers, slated for publication in Interacting with
Computers; "When Bugs Sing," slated for publication in Interacting with
Ultimate memory demoed
makes bugs sing
Nanotubes grown in place
Quantum secrets ride
Chip keeps atoms in line
Research News Roundup
Research Watch blog
View from the High Ground Q&A
How It Works
News | Blog
Buy an ad link