Ten-Hand Piano: A Networked Music Installation

Piano a dez mãos: uma instalação de música em rede

Álvaro Barbosa
Universidade Católica Portuguesa
Escola das Artes
Centro de Investigação em Ciência e Tecnologia das Artes (CITAR)
Rua Diogo Botelho, n.º 1327 4169-005 Porto, Portugal


UCP - Universidade Católica Portuguesa

Retirado de: http://convergencias.esart.ipcb.pt

RESUMO: Este artigo apresenta os últimos desenvolvimentos do sistema de Objetos de Som Públicos (PSOs), um framework experimental para implementar e testar novos conceitos de Networked Music. O projecto de instalação pública interactiva através do sistema PSO foi encomendado em 2007 pela Casa da Música, principal sala de concerto do Porto. Resultou em uma estrutura musical distribuída com até dez terminais de desempenho interativos distribuídos ao longo dos corredores da Casa da Música, controlando coletivamente um piano acústico compartilhado. A instalação permite que os visitantes colaborem remotamente uns com os outros, dentro do edifício, utilizando uma interface de software personalizado desenvolvido para facilitar práticas de música colaborativa e sem requisitos em termos de conhecimento prévio de desempenho musical.

PALAVRAS-CHAVE: Network Music Instruments; Desempenho colaborativo em tempo real; Instrumentos de música eletrônica; Interfaces com Comportamento; Composição Algorítmica; Música Pública; Objetos sonoros.

ABSTRACT: This paper presents the latest developments of the Public Sound Objects (PSOs) system, an experimental framework to implement and test new concepts for Networked Music. The project of a Public interactive installation using the PSOs system was commissioned in 2007 by Casa da Musica, the main concert hall space in Porto. It resulted in a distributed musical structure with up to ten interactive performance terminals distributed along the Casa da Musica's hallways, collectively controlling a shared acoustic piano. The installation allows the visitors to collaborate remotely with each other, within the building, using a software interface custom developed to facilitate collaborative music practices and with no requirements in terms previous knowledge of musical performance. Keywords Network Music Instruments; Real-Time Collaborative Performance; Electronic Music Instruments; Behavioral Driven Interfaces; Algorithmic Composition; Public Music; Sound Objects.


KEYWORDS: Network Music Instruments; Real-Time Collaborative Performance; Electronic Music Instruments; Behavioral Driven Interfaces; Algorithmic Composition; Public Music; Sound Objects.

1. Introduction

The Public Sound Objects (PSOs) project consists of the development of a networked musical system, which is an experimental framework to implement and test new concepts for on-line music communication. It not only serves a musical purpose, but it also facilitates a straight-forward analysis of collective creation and the implications of remote communication in this process.
The project was initiated in 2000 [1] [2] at the Music Technology Group (MTG) from the Pompeu Fabra University in Barcelona, and most developments since 2006 have been undertaken by the Research Center for Science and Technology of the Arts (CITAR) at the Portuguese Catholic University in Porto.
The PSOs system approaches the idea of collaborative musical performances over a computer network as a Shared Sonic Environment aiming to go beyond the concept of simply using computer networks as a channel to connect performing spaces. It can run entirely over WWW, and its underlying communication protocol (Hypertext Transfer Protocol - HTTP), in order to perform over a regular Internet Connection and achieve the sense of a Public Acoustic Space where anonymous users can meet and be found performing in collective Sonic Art pieces.
The system itself is an interface-decoupled Musical Instrument, in which a remote user interface and a sound processing engine reside with different hosts, given that it is possible to accommodate an extreme scenario where a user can access the synthesizer from any place in the world using a web browser.
Specific software features were implemented in order to reduce the disruptive effects of network latency [3], such as dynamic adaptation of the musical tempo and dynamics to communication latency measured in real-time.
In particular, the recent developments presented in this paper, result from a commission in 2007 of an Interactive Sonic Art Installation form Casa da Musica, the main concert hall space in Porto. The resulting Setup is a distributed musical structure with up to ten interactive performance terminals distributed along the Casa da Musica’s hallways, collectively controlling a shared acoustic piano.
It Includes:

  • The adaptation of the Original synthesizer (a Pure-Data [4] sound Engine) to a Yamaha Disklavier Piano [5]
  • Redesign of the interactive sound paradigm in order to constructively articulate multiple instances of experimental users to an ongoing musical piece in real time.
  • Introduction o an Ethersound [6] acoustic broadcast system for the clients musical feed-back
  • Design of a physical infrastructure, coherent with the Casa da Musica architecture, to support the client and server terminals.


2. Background Topics

2.1 Sound Objects

Community-driven creation, results in a holistic process, i.e., its properties cannot be determined or explained by the sum of its components alone [7]. A community of users involved in a creation process, through a Shared Sonic Environment, definitely constitutes a Whole in Holistic sense.
According to Jan Smuts (1870-1950), the father of Holism Theory, the concept of a Whole implies its individual parts to be flexible and adjustable. It must be possible for the part to be different in the whole from what it is outside the whole. In different wholes a part must be different in each case from what it is in its separate state.
Furthermore, the whole must itself be an active factor or influence among individual parts, otherwise it is impossible to understand how the unity of a new pattern arises from its elements. Whole and parts mutually and reciprocally influence and modify each other.
Similarly, when questioning object’s behaviors in Physics it is often by looking for simple rules that it is possible to find the answers. Once found, these rules can often be scaled to describe and simulate the behavior of large systems in the Real World.
This notion applies to the Acoustic Domains through the definition of Sound Objects as a relevant element of the music creation process by Pierre Schaeffer in the 1960’s. According to Schaeffer, a Sound Object is defined as:

Any sound phenomenon or event perceived as a coherent whole (…) regardless of its source or meaning” (Schaeffer, P., 1966).

Sound Object (I’object sonore), refers to an acoustical object for human perception and not a mathematical or electroacoustical object for synthesis. One can consider a sound object the smallest self-contained particle of a Soundscape [8]. Defining a universe of sound events by subsets of Sound Objects is a promising approach for content-processing and transmission of audio [9], and from a psychoacoustic and perceptual point of view it provides a very powerful paradigm to sculpt the symbolic value conveyed by a Soundscape.
In an artistic context the scope for the user’s personal interpretation is wider. Therefore such Sound Objects can have a much deeper symbolic value and represent more complex metaphors. Often there is no symbolic value in a sound, but once there is a variation in one of its fundamental parameters it might then convey a symbolic value.
All these ideas about Sound Objects and the Holistic nature of community music are the basis for the main concept behind the Public Sound Objects System. In fact, in PSOs raw material provided for each user, to create his contribution to a shared musical piece, is a simple Sound Object. These Sound Objects, individually controlled, become part of a complex collective system in which several users can improvise simultaneously and concurrently.
In the system a server-side real-time sound synthesis engine (a Disklavier Piano in the case of the Casa da Musica installation) provides an interface to transform various parameters of a Sound Object, which enables users to add symbolic meaning to their performance.


2.2 About Networked Music

In his Keynote Speech from ICMC 2003 Roger Dannenberg mentioned “Networked Music” as one of the promising research topics and at least four papers [2], [10] and [11] were centered on this topic, even though they were scattered over different panels, instead of one distinct session.
Since then the term Networked Music has become increasingly consensual in defining the area, and according to Jason Freeman’s definition [12]: it is about music practice situations where traditional aural and visual connections between participants are augmented, mediated or replaced by electronically-controlled connections.
In order to have a broad view over the scientific dissemination of Networked Music research I present some of the most significant Landmarks in the field over the last decade:



2.2.1 Summits and Workshops

The ANET Summit (August 20-24, 2004)

The summit was organized by Stanford University’s Center for Computer Research in Music and Acoustics (CCRMA) and held at the Banff Center in Canada, was the first Workshop event addressing the topic of High quality Audio over Computer Networks. The guest lecturers were Chris Chafe, Jeremy Cooperstock, Theresa Leonard, Bob Moses and Wieslaw Woszczyk. A New edition of the ANET Summit is planed for April 2008
The Networked Music Workshop at ICMC (September 4, 2005).
This Workshop was held in Barcelona and resulted from experience in previous ICMCs, which called for the need to realize such an event. Guest Lecturers were: Álvaro Barbosa (Pompeu Fabra University, MTG), Scot Gresham-Lancaster (Cogswell College Sunnyvale, CA), Jason Freeman (Georgia Institute of Technology), Ross Bencina (Pompeu Fabra University, MTG).



2.2.2 PhD Dissertations

These are some relevant dissertations published on the topic:
2002 Golo Föllmer “Making Music on the Net, social and aesthetic structures in participative music” [13]; 2002 Nathan Schuett “The Effects of Latency on Ensemble Performance” [14]; 2003 Jörg Stelkens “Network Synthesizer” [15]; 2003 Gil Weinberg “Interconnected Musical Networks: Bringing Expression and Thoughtfulness to Collaborative Music” [16]; 2006 Álvaro Barbosa “Displaced Soundscapes” [17]


2.2.3 Journal Articles

There is a number of Survey and partial overview articles on the topic of Networked Music [18], [19], [20] [21] and [22] however a special issue of the journal Organised Sound from 2005 [23], edited by Leigh Landy, specifically focused on the topic of Networked Music and includes many of the relevant references in this area.



3. The PSOs Installation

Casa da Musica is the main concert venue in the city of Porto, and it has a strong activity in what concerns contemporary and experimental forms of Music. The commission for the Public Sound Objects Installation had the underlying idea of bringing music to the hallways of the house of music, so that the visitors could actually interact with it.


Fig.1 – Casa da Musica Building1


The final implementation consists of a Disklavier Piano controlled via MIDI by s server that simultaneously can be used as a terminal, located at the main foyer of Casa da Musica. This server accepts incoming control data generated by 10 client computers located in diverse points of the hallways of a scenic route of the building. Incoming data is transmitted over the building’s IP Network using Open Sound Control [24].


Fig. 2 – The PSOs Server connected to the Disklavier Piano and two of the clients which remotely control the same Piano



The sound generated at the Servers site conveys the overall performance of every user and is streamed back to each client using an ETHERSOUND [6] system, which produces latencies under 100 ms on the building’s LAN.


Fig. 3 – A PSOs Client with the ETHERSOUND Hub, Speakers and Keyboard concealed on the structure.



3.1 The User Interface

The graphical user interface is based on a bi-dimensional graphical metaphor of an ever-going bouncing ball enclosed in a square shape box. Each time the ball hits one of the walls a piano key is triggered by the server according to a pitch defined by the value of a stylized fader that frames the Box (each fader determines the pitch of a sound triggered in its adjacent wall).


Fig. 4 – PSOs Client interface showing the representation of 5 users


Each of the clients actuating at a given moment are visually represented in real-time by grey balls while the user himself controls a distinctive orange ball. The user can also add a trail to his ball producing an arpeggio sound (or a chord if the trail extension is zero), given that the scale of notes each client can produce was anticipated to create a harmonic soundscape when different sound overlap in time.
The PSOs system integrates several features to overcome Network Latency issues already published in [3]. Nonetheless, in this version a new Latency tolerance feature was implemented to improve the perceptive correlation between an impact and a triggered sound, using a simple sound panorama adjustment at the sound server and consequently adding consistent sound panning with the object’s behavior at the graphical user interface.


Fig. 5 – Representation of Impacts VS Triggered Sound with sound panorama adjustment in the presence of latency (Dt)


The basic idea consists of only transmitting a sound object trough the Right Channel of the streamed Soundscape stereo mix, when a ball hits the right wall, transmitting only through the Left Channel when a ball hits the left wall and transmitting in booth channels (L+R) if the ball hits the top or bottom wall.
Sound Panorama Adjustment adds an extra cue to perception in temporal order of triggered Sound Objects and respective correlation to ball impacts.



4. Conclusions and Future Work

The PSOs Installation at Casa da Musica allows a piano to be controlled by 10 instances simultaneously (Ten Hands!) in a coherent and constructive manner, which would hardly be possible to do in a traditional way.
Even though the interface is radically different than the normal control paradigm of a piano it is based on the same fundamental musical facets (Rhythm, Pitch, Timbre and Dynamic) and therefore it is an engaging experience, since the users recognize a familiar result achieved trough a totally different way.
The interface is simple enough to achieve a musical soundscape with zero learning time and without any previous musical practice experience, which made the system very accessible and popular for the average 500 daily visitors of the Casa da Musica.
Controlling a popular acoustical instrument brings the users closer to the musical experience and in this sense we would like to further develop this system adding a pool of instruments to the piano, such as wind, string and percussion instruments controlled by Robotics.



5. Acknowledgments

The author would like to thank the people that collaborated in this project: Jorge Cardoso (UCP), Jorge Abade (UCP) and Paulo Maria Rodrigues (Casa da Musica).



1 Image Source “House of Music Opening Day” Wikipedia Commons under the license GFDL (GNU Free Documentation License).


Bibliographic references

[1] Barbosa, A. and Kaltenbrunner, M. Public Sound Objects: A shared musical space on the web. 2002. IEEE Computer Society Press. Proceedings of International Conference on Web Delivering of Music (WEDELMUSIC 2002) - Darmstadt, Germany

[2] Barbosa, A., Kaltenbrunner, M. and Geiger, G. Interface Decoupled Applications for Geographically Displaced Collaboration in Music. 2003. Proceedings of the International Computer Music Conference (ICMC2003)

[3] Barbosa, A., Cardoso, J. and Geiger, G. Network Latency Adaptive Tempo in the Public Sound Objects System. 2005. Proceedings the International Conference on New Interfaces for Musical Expression (NIME 2005); Vancouver, Canada.

[4] Puckette, M. Pure Data. 269-272. 1996a. International Computer Music Association. Proceedings of the International Computer Music Conference, San Francisco (ICMC96)

[5] Yamaha Disklavier Piano: http://www.yamaha.co.jp/english/product/piano/product/europe/dl/dl.html (Cunsulted 2008/01/30)

[6] ETHERSOUND: http://www.ethersound.com/ (Cunsulted 2008/01/30)

[7] Smuts, J. Holism and Evolution. 1926. Macmillan, London UK

[8] Schaeffer, P., Traité des Objets Musicaux., Le Seuil, Paris, 1966

[9] Amatriain, X. and Herrera, P. Transmitting Audio Content as Sound Objects. 15-6-2002. Proceedings of the AES22 International Conference on Virtual, Synthetic and Entertainment Audio

[10] Stelkens, J. peerSynth: A P2P Multi-User Software with new techniques for integrating latency in real time collaboration. 2003. Proceedings of the International Computer Music Conference (ICMC2003)

[11] Obu, Y., Kato, T. and Yonekura, T. M.A.S.: A Protocol for a Musical Session in a Sound Field Where Synchronization between Musical Notes is no garanteed. 2003. International Computer Music Association. Proceedings of the International Computer Music Conference (ICMC2003), Singapore

[12] Freeman, J. The Networked Music Workshop at ICMC 2005, Barcelona (September 4, 2005)

[13] Föllmer, G. 2002 Making Music on the Net, social and aesthetic structures in participative music. Ph.D. Thesis, Martin Luther Universität Halle-Wittenberg – Germany

[14] Schuett, N. 2002 The Effects of Latency on Ensemble Performance. Ph.D. Thesis, Stanford University, California – USA

[15] Stelkens, J. 2003 Network Synthesizer. Ph.D. Thesis, Ludwig Maximilians Universität, München – Germany

[16] Weinberg, G. 2003 Interconnected Musical Networks – Bringing Expression and Thoughtfulness to Collaborative Music Making. Ph.D. Thesis, Massachusetts Institute of Technology, Massachusetts – USA

[17] Barbosa, A. 2006 Displaced Soundscapes: Computer Supported Cooperative Work for Music Applications. Ph.D. Thesis, Pompeu Fabra University, Barcelona – Spain

[18] Sergi Jordà, S. 1999 Faust Music On Line (FMOL): An approach to Real-time Collective Composition on the Internet, Leonardo Music Journal, Volume 9, pp.5-12

[19] Tanzi, D. 2001 Observations about Music and Decentralized Environments, Leonardo Music Journal, Volume 34, Issue 5, pp.431-436

[20] Barbosa, A. 2003 Displaced Soundscapes: A Survey of Network Systems for Music and Sonic Art Creation, Leonardo Music Journal, Volume 13, Issue 1, pp.53-59

[21] Weinberg, G. 2005 Interconnected Musical Networks: Toward a Theoretical Framework, Computer Music Journal, Vol. 29, Issue 2, pp.23-29

[22] Traub, P. 2005 Sounding the Net: Recent Sonic Works for the Internet and Computer Networks, Contemporary Music Review, Vol. 24, No. 6, December 2005, pp. 459 – 481

[23] Landy, L. 2005 Organised Sound 10 (Issue 3), Cambridge University Press, U.K. (OS: ISSN: 1355-7718)

[24] Wright, M. and Freed, A. 1997 Open Sound Control: A New Protocol for Communicating with Sound Synthesizers, proceedings of the International Computer Music Conference

[25] Nella, M. J. Constraint Satisfaction and Debugging for Interactive User Interfaces. Ph.D. Thesis, University of Washington, Seattle, WA, 1994.

Reference According to APA Style, 5th edition:
Barbosa, . (2008) Ten-Hand Piano: A Networked Music Installation. Convergências - Revista de Investigação e Ensino das Artes , VOL I (2)