Intelligent Musical Instruments: future of musical performance or the demise of the performer?

INTERFACE Journal for New Music Research, The Netherlands, December 1993.

by W. Andrew Schloss and David A. Jaffe

ABSTRACT: This paper examines the potential problems that "too much" technology in musical performance can create. In developing very powerful computer-assisted instruments, and in decoupling the sound production from the gesture, issues of what performance is really about start to surface. This is a relatively recent problem, because it is only in the last few years that realtime performance has been widely possible in computer music. As a case in point, we will discuss a co-composed piece entitled "Wildlife," that involves many of the critical issues.


The history of virtuosity in music is an interesting one. Virtuosity is not a given in all musical traditions. Western culture, with its emphasis on the individual, is much more centered on individual accomplishments than many other cultures are. It is not pure ethnocentricity that causes us to say "the music of the Yoruba" in the same way we might refer to "the music of Beethoven" as a body of work to be enjoyed or studied. The fact that a body of work by a single composer could be regarded in any way equivalent to the work of an entire people shows a great difference in cultural approach. Technology has changed our concept of "performance," but we retain our fascination with virtuosity in general.

Concert music is still a vital part of our musical life. To ensure that it remains so in the decades to come, despite drastic changes in style and technological resources, we must reflect on why people go to concerts at all. We suggest that one of the significant aspects of live performance (from the point of view of the audience) is virtuosity, and that there is a danger that perception of virtuosity may erode as new technologies are applied to musical performance.

The Problem

In order to increase the amount of information that can be conveyed between musician and instrument ("bandwidth"), we can use a computer to make the instrument able to respond in ways that appear to be "intelligent." This means developing algorithms that interpret gestures from the performer and act upon them in complex ways. However, a new problem emerges here: whereas acoustic instruments have exhibited since the beginning of time a nearly one-to-one correspondence between the performer's action and the sonic result, these new instruments, with their invisible technology (seeming to border, at times, on "magic"), have no such intrinsic relation. The question becomes: do we need a perceivable cause-and-effect relationship in live performance?

We believe this is a question still to be answered, and in our own work we keep it in mind at all times. An amusing example of this problem was observed when the Flying Karamozov Brothers, a theatrical and juggling troupe, had wireless MIDI transmitters mounted in hockey helmets, with which they actually played music by juggling and repeatedly bashing each other on the head with their juggling clubs in perfect synchrony. Even though the audience was told in advance that they were actually performing music by juggling and hitting themselves on the head (triggering MIDI events), the audience thought they were playing a tape, and juggling to the tape. They had reached the threshold of "magic" and were doing a lot of work for nothing.

An example: The Radio Drum and the MAX real-time processing system

The Mathews/Boie Radio Drum is an interface that is sensitive to location in three dimensions, while maintaining a high degree of temporal accuracy. When combined with MAX (or another similar real-time processing system, such as the Music Kit), the composer can make "virtual" configurations of the drum, so that a particular event or combination of events can have practically any musical result. This combination of a real-time configurable application with a flexible controller makes an ideal meta-instrument for high-level control of synthesizers or samplers in a variety of performance situations.

Let us group the possible interaction schemes into three categories, the timbral (microscopic) level, note (middle) level and formal (macroscopic or process) level. We will discuss these levels as they relate to performance.

The timbral level, typically involves continuous control over some aspect of an ongoing sound. This is the familiar micro-control that a performer exerts over an acoustic instrument within each note, producing such effects as vibrato, tremolo, crescendo, and other expressive nuances. An early example of an electronic instrument that allows for this sort of control is the Theremin, in which a hand moving through space alters pitch continuously. Controlling sound on the timbral level typically requires great skill, but it is usually very clear what the intention of the player is (eg vibrato or glissando)–there is little danger of the audience's losing site of the meaning of a performer's gesture. Unfortunately, most current controllers do not always fare well in this domain due, primarily due to the influence of the keyboard-orientation of MIDI. The Radio Drum, however, is quite well-suited for this due to its 3-dimensional control.

The note level corresponds with what is on the printed page of music. Each note on the page corresponds to a note that is sounded. Similarly, each gesture on the electronic instrument has a perceptible one-to-one causal relationship between physical action and sonic result. Here, an electronic keyboard, for example, is practically equivalent to a conventional piano, and various percussion controllers are also very easy to understand. Again, the audience generally has little problem equating the performed gesture with the acoustic result.

More problematic is what could be called the "process" level. Process as a concept in music has been discussed throughout music history. Examples include the early isorhythmic motets, the fugue and the twelve-tone manipulations of the 20th century. Process itself became a particular focus in the 1960's in the music of Chadabe, Reich and others. Nevertheless, only in the last ten years has the power to control high level computer processes in realtime become commonly available to composers and performers. Realtime digital sound processing and flexible real time music systems (like MAX and the Music Kit) have greatly increased the power of the player, but have simultaneously exacerbated the issues of clarity in performance. "Hyperinstruments," "Virtual Instruments" and "Intelligent Instruments" are but three names that are being applied

to describe this phenomenon. No longer can the performer assume the audience will be able to make any connection whatsoever between his gesture and the audible result.

Future Education and Training for Young Musicians

These issues will have a profound effect on musical training as intelligent instruments are studied by children who may never have played the acoustic version of the instrument. We consider a few anecdotal examples.

One of the first electronic instruments was the electric guitar. Originally, the electric guitar was merely an amplified version of its acoustic relative. However, as performers such as Jimi Hendrix in the 1960's extended its expressive vocabulary through the use of distortion electronics, time-varying filters and other devices, the instrument took on a sound, technique and repertoire of its own, increasingly separate from that of the acoustic guitar. There are now electric guitar teachers that are quite different from acoustic guitar teachers. But despite these changes, the electric guitar is still an instrument in which there is a direct acoustic (if amplified) coupling between the performer's gestures and the acoustic result.

A far different situation arises with the new guitar controllers. These are not really guitars at all, but general purpose controllers in the shape of a guitar, thus enabling a player to use his learned experience as a guitarist. These controllers have no intrinsic sound of their own--there is no transducer or other amplifying device. In fact, there need not even be any strings! The sound production is entirely decoupled from the instrument itself.

Another example comes from our own musical backgrounds. Schloss was trained as a percussionist, and that training is implicitly available to him even as he plays a new instrument that has no acoustic behavior at all. Jaffe has a similar history with the acoustic violin and mandolin. But what would it be like if we never learned to play real drums or stringed instruments, only the virtual electronic versions? We would probably lack sufficient intuition and nuance to become effective performers on these new instruments. Another example is the MIDI wind controllers, which are vastly less powerful than real instruments in terms of nuance of control. Even though they may be connected to powerful synthesizers, they tend to be less expressive than their acoustic counterparts. In particular, they have limited refinement in their control of pitch, timbre, and dynamics.

Some of these problems are a result of the impoverishment of the MIDI specification itself. Yet it seems that learning on even an improved wind controller, without any previous experience with a real wind instrument, could deprive a young student of a great deal of musical intuition. Finally, there is the desire on the part of some computer music researchers to create new instruments that are intelligent in a different way: these instruments should "know" what to do and require very little skill to play. Though this might allow beginners and amateurs to enter into music making quickly and enjoyably, this trend might ultimately limit a person's understanding of music. The question is whether these instruments will stimulate due to their immediate accessibility, or suffocate due to a kind of atrophy of learned musicality. It is an open question at this point.

Wildlife–A Case Study

Wildlife is a computer-extended duo in five movements for Mathews/Boie Radio Drum and Zeta violin. It was co-composed by the authors and was premiered by the composers in Victoria, Canada in 1991. It has been performed numerous times in North America and Europe and is soon to be released on an upcoming CDCM Centaur Series: The Virtuoso in the Computer Age , Volume 15, a compact disc of music featuring the Radio Drum.

The name "Wildlife" is used in two senses. First, it refers to the improvisational nature of the work. All materials are generated in direct response to the performers' actions and there are no pre-recorded or stored sequences. Furthermore, the malleable nature of the instruments allows the traditional boundaries that separate one instrument from another to be broken down. As a simple example, the violinist's glissando may change the pitch of chords played by the percussionist. Allowing the computer a degree of autonomy takes the performers a further step away from the customary ensemble relationship. Thus, they find themselves "living on the wild side."

The autonomy provided the computers gives rise to the second sense in which the name "Wildlife" applies. Robotic expert Hans Moravec describes, in his recent work Mind Children, a world in which autonomous artificial life forms breed, propagate, compete and interact. These life forms can be beneficial, parasitic or benign. In Wildlife, the computers spawn independent processes that suggest such beings.

Technical Description

As described earlier, the Mathews/Boie Radio Drum is a sensor capable of reporting accurately the position of two mallets in three dimensions. It generates no sound; the effect of a performed gesture is entirely determined by software. The Zeta violin is a solid-body electric violin with both a MIDI output and an amplified electronic sound. Each string has its own pickup and pitch detector, allowing for independent pitch bend for each string.

The system configuration consists of both the Zeta violin and the Drum passing information to a Macintosh IIci computer, which does preliminary gestural processing and passes MIDI information to a SampleCell sample player and a NeXT computer. The NeXT does further gestural processing and algorithmic music generation, performs synthesis on the NeXT's built-in DSP chip, and sends MIDI to a Yamaha TG77 synthesizer. The software on the Macintosh is based on the Max system, while the software on the NeXT is based on Ensemble and the NeXT Music Kit.

In addition to the Radio Drum and the Zeta violin, both performers have a number of foot pedals and switches. Of particular interest is the percussionist's 18-key chromatic bank of organ-style velocity-sensitive foot pedals. These pedals are used for numerous kinds of control information rather than playing pitches, however, we have found that the standard "black and white key" orientation facilitates locating the footswitches quickly and easily.

Plan of Attack

We began work on the piece by exploring a wide range of interactive scenarios. We quickly found that many were unsatisfactory for various reasons. In particular, as improvising performers, we found it essential to feel that our actions had a discernable and significant effect on the music being produced. In many situations, one or the other of us felt that his influence was inconsequential. The situation was analogous to playing a solo in a jazz ensemble with insensitive accompanists. The manner in which we felt we needed to be able to exert influence could be very different for each of us in a given situation. Yet, there was definitely an intangible but undeniable difference between those situations where we felt our improvisational imagination was fired and those where we felt it inhibited by opaque complexity.

Visceral Learning

It was difficult to predict whether or not an interactive scenario imagined on paper would turn out to be effective in performance. The only way we could decide what worked well was by long hours performing with each setup, searching for material that seemed to complement it, and exploring its potential. Though we both understood fully the logic being executed by the computer programs, it was only by exploration as performers that we discovered hidden aspects. Thus, the music for each of the five movements was developed in parallel with the interaction scheme.

As a simple example, in the fourth movement, the violinist supplies the pitches that make up the percussionist's improvisation. The percussionist can choose to play recently-played pitches or can go back in time to pitches played earlier. In this context, the violinist plays only occasionally and in such a manner as to change the flow of the ongoing music. This movement was particularly difficult for him because the effect of material he played was evident only some time later when the percussionist played these pitches. Yet, an implementation detail turned out to supply the answer. It turns out that the "remembered" pitches played by the violinist are stored in a buffer that is not circular. Thus, every hundred notes (this number was at first set arbitrarily), the buffer would be empty and would start to be refilled again. This quirk turned out to provide just the "foot in the door" that the violinist needed. By playing tremolo, he could fill up the whole buffer with a single pitch and constrain the percussionist to that pitch. The implementation also guaranteed that every now and then the percussionist would be forced to play only very-recently performed pitches. Thus, what started out as an arbitrary irrelevant constraint turned out to be an asset in disguise. "It's not a bug it's a feature!"

Musical Cowboys

When we gave the computer a large degree of autonomy, the major problem became how to avoid the feeling the music was "getting away from us." One way to deal with this problem is to consider the performer's role as analogous to that of a conductor of a piece that doesn't use strict rhythmic coordination between parts. In such a piece, the conductor gives signals that control the large-scale flow of the music, but without specifying the individual details. To use a more colorful analogy, the independent computer processes are like cattle that are allowed to wander over the open plains and the performer's control is that of the cowboy who reigns them in when it's time to go into the corral.

As an example, in the third movement, the computer is generating melodic material based on the pitches played by the violinist, but transposing those pitches to any octave and using fractal shapes to derive melodic, rhythmic and dynamic contours. The percussionist can exert control over this process by changing the upper and lower range of the computer process, by changing whether or not repeated notes are tied, and by changing the tempo. The computer process also can be made to follow players' dynamics. The violinist can change the density of the computer process by playing repeated notes. Thus, the computer processes can be allowed to wander freely and then be reigned in suddenly in response to the performer's actions. Nevertheless, the process is interesting and complex enough that the computer often seems to have a mind of its own. We are never sure how it is going to behave and are often surprised (and sometimes perplexed) by its seeming whimsy.

Crossing Boundaries

Traditional instruments have clear boundaries. They may play in unison, combining to produce a new timbre; they may combine harmonically or contrapuntally. But each performer is in control of the sound produced by his instrument and is the sole determiner of the notes he will play and when he will play them. In contrast, with "virtual instruments" (controllers) like the Radio Drum and controller-instrument hybrids like the Zeta Violin, the traditional boundaries between performers can become like permeable membranes. As a simple example, the violinist's glissando can change the pitch of notes produced by the percussionist and the percussionist can control the loudness of the electric violin sound. This situation is analogous to the humorous trick performed by bluegrass

bands in which the guitarist reaches his left arm around the banjo player and plays the chords of the banjo while the banjo player does the picking, and at the same time, the banjo player reaches his left arm around the guitarist's back and plays the chords for his instrument.

The crossing of boundaries can happen on a larger formal level as well. In the fifth movement, the percussionist plays arpeggiated consonant chords. The root of each chord can be specified by either the percussionist or the violinist. The violinist specifies the root by stepping on a pedal. The next note he plays sets the new root. If he uses the pedal only occasionally, the harmony changes slowly and is consonant. However, if he uses the pedal frequently, on every note for example, the effect is much more complex and dissonant. Since the percussionists' chords are transposed according to the octave of the violin note, the violinist also has control over the percussionist's range. At the same time, the percussionist has a similar pedal and can surprise the violinist by changing the harmony out from under him, but in a way intimately related to what the violinist has just played. Thus, the harmony emerges as a result of a complex improvisational interaction.

An Example In Detail

The first movement begins with a simple interaction scheme, allowing the audience to perceive the causality between performed action and resulting synthesizer sound. The violin, in addition to its acoustic sound, produces pitches via a "chord mapping set", defined as twelve "chord mappings." A chord mapping is a chord that is produced when a particular pitch class is performed and transposed in a manner corresponding to the performed octave. The Drum selects which of several chord mapping sets is active. As an example, one set might produce chords derived from chromatic tone clusters while another might produce a different octave-displacement for each pitch.

The Drum's horizontal axis controls register, the vertical axis controls duration and the height above the surface controls loudness. The surface is also partitioned in half, with one part of the Drum playing chords, and the other playing single notes. Overlaying this partition is a grid that the percussionist uses to select the active chord mapping set. Thus the familiar gesture of striking the drum can have the unfamiliar result of changing the harmonization of the violinist's melody, an effect usually considered in the realm of composition rather than performance.

Another interesting aspect of this movement from an ensemble standpoint is that both performers are playing the same synthesized sound at the same time, resulting in ambiguity as to who does what and enabling one player to "pull the rug out from under" the other player.


Our experience with Wildlife shows that improvisational ensemble music and interactive instruments can be a powerful combination. The traditional inviolability of a performer's sole control over his instrument can be relaxed and the degree of invasion of one performer's control over the other's instrument can be controllable as a musical parameter. Adding to this situation semi-autonomous computer processes that the players can control in the manner of a conductor further enriches the environment.

However, to discover an effective interactive scenario, it is necessary to spend a good deal of time playing with the system and learning its idiosyncrasies. Improvisation is ideal for allowing this to occur, since it lets the performer react spontaneously to the musical situation. No amount of programming skill and cleverness is a substitute for the process of using the system in a musical context.


Though the power we now have in computer music is wonderful, exhilarating and open-ended, and though it frees us forever from the tyranny of the tape machine, we have entered an era in which cause-and-effect, an inherent aspect of musical performance since the beginning of time, is suddenly evaporating.

Digital signal processing will help a great deal in this problem, because the virtuosity inherent in playing acoustic instruments can be retained. As for the global problem of complexity and loss of the perception of cause-and-effect, We believe that this is a problem that must be dealt with individually in every situation, and to some extent will be answered by the response of the audience.


Thanks to visionary instrument builders Max Mathews and Bob Boie (Radio Drum) and Keith Mcmillian (Zeta violin.) Thanks also to Michael McNabb, whose Ensemble program has been enormously useful in this work, Julius Smith, who co-developed the NeXT Music Kit with the author, and Miller Puckette and David Zicarelli, whose Max program proved extremely valuable. [Portions of this paper first appeared in the proceedings of the First International Workshop on Man-Machine Interaction in Live Performance, sponsored by the CNUCE/CNR, Pisa, Italy, and the proceedings of the 1992 ICMC, San José, California.]



Boie, R. A., L.W. Ruedisueli and E.R. Wagner "Gesture Sensing via Capacitive Moments" Work Project No. 311401-(2099,2399) AT&T Bell Laboratories, 1989.

Jaffe, D. A and W. A. Schloss "The Computer-Extended Ensemble" Computer Music Journal (Forthcoming)

Jaffe, D. A and W. A. Schloss "Wildlife," an interactive duo for Mathews/Boie Radio Drum and Zeta Violin. CDCM Series Compact Disc, Volume 15, The Virtuoso in the Computer Age, on Centaur Records. Available Spring 1993.

Jaffe, D. "Ensemble Timing in Computer Music." 1985. Computer Music Journal, MIT Press, 9(4):38-48.

Jaffe, D. "The Computer-Extended Ensemble." LULU, Buneos Aires, Argentina, Autumn 1992.

Chadabe, Joel Interactive Composing: An Overview Computer Music Journal 8:1 22-28, 1984.

Mathews, M. V. and W. A. Schloss "The Radio Drum as a Synthesizer Controller." Proceedings of the 1989 ICMC, Columbus Ohio.

Mathews, M. V. "The Conductor Program and Mechanical Baton" Current Directions in

Computer Music Research, MIT Press, 1988.

McNabb, M. "Ensemble, an Interactive Performance and Composition Environment."

Proceedings of the 1990 Audio Engineering Society Conference, L.A.,CA.

Moravec, H. Mind Children. Harvard University Press, 1988.

Puckette, Miller "Amplifying Musical Nuance" J. Acoust. Soc. of Am. Suppl. 1, Vol. 87,

Spring, 1990.

Reich, Steve Writings About Music The Press of the Nova Scotia College of Art and Design, Halifax, N.S. Canada, 1974.

Smith, J. and D. Jaffe and L. Boynton. "Music System Architecture on the NeXT Computer." Proceedings of the 1989 Audio Engineering Society Conference, L.A., CA.

Schloss, Andrew "Recent Advances in the Coupling of the Language Max with the Mathews/Boie Radio Drum" ICMC Glasgow 1990 Proceedings.

Wessel, David "Computer-Assisted Interactions Between Two Musicians" J. Acoust. Soc. of Am. Suppl. 1, Vol. 87, Spring, 1990.

Zicarelli, David "Writing External Objects for MAX" Opcode Systems, 1991.