Variations
An Exploration of the Music of Jan MensaertThis article appeared on the CD-ROM that accompanied the exhibition. It describes the sculpture that was exhibited at the Storm King Music Festival in New York during July, 2000. A new interactive musical sculpture was created for the final exhibition in Belgium.
Abstract
The idea for "Variations" began when I was asked to provide a musical recording of a short piece written by Jan Menseart, the Belgian poet. The music was to be part of a retrospective exhibition of his work, including drawings, photographs and writings. As I pondered the music, the idea came to me to create a piece that was more than simply music. This would be something with a universal approach, containing the elements of music, images and an art object that together represent a creative interpretation of the original music. "Variations" can best be described as a digital art installation. There are three elements to the piece. The first is a computer on a pedestal, which has an interactive CD-ROM containing several musical variations of Jan Menseart's music. Along with the music, the CD-ROM has abstract visual interpretations of the music, a selection of Jan Mensaert's images/photographs, an artist statement, project description and biography. The second element of the installation is a rectangular sculpture approximately 40" wide and 36 " high, consisting of ten horizontal threaded rods with a nut and washer element spaced at intervals that relate to the musical composition. The third element is a simple music stand with the original sheet music on it. The installation is meant to encourage the viewer to experience a single piece of music in a variety of ways. The music stand is a reminder of the role of the musician in the creation of the music and a frame of reference when one looks at the sculpture, a comparison between the literal and the abstract. The role of the computer in the piece is to provide a "virtual musician" that performs the various interpretations of the music.Theoretical Background
As I conducted my research about Jan Mensaert, I learned that he was a person who had a wide range of interests and I thought that an art piece that had more than just a single interpretation of the music was an exciting challenge and in line with his world view. The following quotes from an essay I recently published in LEONARDO will give some insight into why I chose to create art the way I do. "I very often reread books that have made an impression on me. Several years ago, while on my way to the SIGGRAPH 94 computer graphics conference, I brought along a copy of Magister Ludi (also published under the title The Glass Bead Game) by Hermann Hesse to read on the plane. The following quotes stood out from the rest of the text and prompted me to stop and reevaluate my creative work and the field of digital art in general." (1)"In the formal game, the player sought to compose out of the objective content of every game, out of the mathematical, linguistic, musical and other elements, as dense, coherent, and formally perfect a unity as possible..."
"(O)ne can be a musician or Glass Bead Game player and at the same time wholly devoted to rule and order. The kind of person we want to develop, the kind of person we aim to become, would at any time be able to exchange his discipline or art for any other." - Hermann Hesse, Magister Ludi (2)
"Having a background as a musician, producer, photographer and visual artist, I had long thought of myself as a Glass Bead Game player. As digital art continues to become the new art form of the next millennium, I am now even more convinced that Hesse's remarks were indeed prophetic. (After all, this book did win him the Nobel Prize for Literature in 1943.)
When I reread Hesse's novel, I was in the process of curating the Second Annual New York Digital Salon, and it struck me that digital art has a Glass Bead Game component to it. In the intervening years, there have been considerable advances in digital imaging, animation software, digital audio/video and the global explosion of the Internet. Access to this technology has become almost ubiquitous and artists are now frequently starting to branch out into media that were not their original primary focus: painters are creating Web sites with interactive and textual components, while photographers are experimenting with video and 3D animation. The internet is also acting as a means of leveling the playing field for artists. It is an inclusive medium that allows text, images, sound and video to be contained within a single artist's Web site.
This branching out has positive and negative connotations for art. It is allowing artists more creative freedom and giving them access to new tools, but it runs the risk of enabling individuals to create work in media in which they do not have a well-developed vocabulary or aesthetic foundation. The question then arises as to whether the work created by these artists is a form of the Glass Bead Game or simply a byproduct of the curiosity of the creative imagination.
Despite the current sophistication in hardware and software, the fact remains that the fundamental element of digital art is data. All images, words, sounds, video and text are ultimately reduced to a string of ones and zeros, stored in a digital file on the computer. On the transistor level, the circuit is either closed or open, in a sense coupled or uncoupled. Depending on the type of media stored, when the data is read by the computer it is converted by ASCII code to make it text, image file formats to make it visible, or digital audio formats to make it aural.
Hesse's reference to the "formal game" can also be interpreted as referring to the computer, which makes visual, aural and textual data as "dense, coherent, and formally perfect a unity as possible," not in the symbolic or intellectual sense, but in a practical way - to allow in to be stored and transferred from machine to machine. One of the benefits of digitizing creative work is that it provides artists a way to access and manipulate this data and to reinterpret it within the context of their individual creative work." (1)
Visual Music
Many artists and composers have been fascinated by the relationship between music and art. Artists have made visual interpretations of music and musicians have employed color, images and sound into their performances. Music is very efficacious at evoking moods and often stirs memories, resulting in various thoughts and often suggesting colors. When approaching the relationship between music and art, we can look at several issues: tempo, structure, composition, mathematical elements and creative interpretation. My approach to creating visual music is to experiment with these various components as they relate to the specific musical passage.Tempo can be looked at as the guiding force behind the amount of detail or the density of the image. A slow tempo would suggest large shapes and very deliberate graphic elements. A fast temp would imply the opposite, a fairly dense image with many components. These relate to how the eye looks at an image and how the ear interprets the sound. Fast tempos require a bit more concentration because there are many more notes in the music. Visually, the eye would be moving slowly or quickly across the image.
Structure relates to the compositional forms used by the composer. Popular music is typically divided into verse, chorus and bridge. Symphonies have several sections to them. The musical experience is by nature a temporal one. There is a beginning, middle and end. Taking this further, there may be repeated passages and variations on particular themes. When applying these principles to a visual interpretation of the music, we now can look at repetition of form and characteristics of the music. For example ascending and descending musical passages suggest lines or shapes that are rising and falling. A musical piece that repeats the same phrase throughout the piece would suggest an image that has a repetitive motif in it.
Composition in this context relates to the style of the music and the instrumentation used. In other words, how the composer chooses the tempo and structure and then weaves it into the particular musical composition that they are writing. This can take many forms depending on the cultural background of the composer and their approach to their work. Classical music is very formal and has very distinct structures. Jazz is based on improvisation and the music composition provides a jumping off point for the musicians to read their own interpretation into the music. Taking these last two ideas into a visual interpretation, classical music might suggest a more precise type of image, whereas jazz may suggest a more abstract approach to the image.
While composition is a subjective process, the mathematical interpretation of music is a very analytical one. While leaving the door open to the musical interpretation of the work, the mathematical elements are easily quantified. This has been made easier by the introduction of MIDI (musical instrument digital interface). MIDI was developed in the mid-1980's by synthesizer manufacturers to allow different instruments to talk to each other. MIDI looks at music through several parameters: note on, note off, pitch, velocity and after touch. This is radically different from the way music is stored on tape, CD's or as a digital audio file in the computer. CD and digital audio files are stored in formats that relate to frequency and amplitude. For example, the standard CD format is 16 bit and 44.1 khz. 16 bit means that the resolution of the samples is high and that there are 44,100 samples of the audio taken per second. A digital file of this type gives the listener an exact reproduction of the sound. However, digital audio files do not give much to the artist. What one sees on the screen is a pattern that varies mainly with the amplitude of the music. The illustration provided is a picture of the digital file for Jan Mensaert's musical piece.
This can be used for inspiration, but the digital audio files of widely diverse types of music often look similar. The illustration also shows the MIDI data from the software ProTools. This also gives one a picture of the music, but now provides a visual reference for pitch and duration. There are several other software programs that give a visual depiction of the music. Taking this one step further, there are now software programs that allow one to "play" images in real time on the keyboard, along with the music.
Getting back to the mathematical analysis of a musical composition, MIDI has now made this process much easier. On a basic level, there are a definable number of notes in a musical piece. Their pitch and duration are quantified, along with the velocity and aftertouch, which are how hard the note is struck and how quickly or slowly the note is ended. By gathering this data, the visual artist can now deconstruct the music into a group of numbers which represent the music. Conversely, these numbers can now be manipulated by the artist into an image that carries the fundamental units of the music within it. For example, low notes can be large spheres and high notes can be smaller spheres. Or, low notes can be darker colors, while higher notes can be lighter colors.
The final element to be considered when discussing visual music is creative interpretation. Although much of the above has related to the visual characteristics of the music and the underlying mathematics of the music, creative interpretation is the point where the artist must take liberties with the data to create a work of art that has aesthetic value. A simple visualization of the mathematical or structural elements of a musical piece is not art, as is a rote performance of the music as it is written not "real" music. It is the musician who makes the sheet music turn into the musical experience. It is the artist who must take all of these elements of the music and turn them into art. Visual music can take the many forms that art has available to it. Visual music has been interpreted into sculptures, images and animation.
Continuing from the LEONARDO essay, "I would like to refer to my own creative work as another example of the deliberate use of the synesthetic approach of coupling and uncoupling data and sensory information with meaning as a tool for inspiration. I am currently working on a sculptural piece based on a brief musical passage written by Belgian poet, Jan Mensaert. I have learned that his earliest ambitions were not poetry or the written word, but in architecture, thus providing me with the reason for creating a three-dimensional piece of work. My approach in creating this sculpture will show how I use synesthesia as an inspirational tool.
My first step was to look at the musical phrase in terms of its mathematical content. I counted the number of musical measures and the number and pitch of the notes to develop for myself a mathematical representation of the musical phrase.
My next step will be to translate the score into MIDI (Musical Instrument Digital Interface) data through a performance of the work to allow myself to look more closely at the musical interpretation aspects of the piece. MIDI allows the recording of musical parameters like pitch, duration, velocity and aftertouch. These last two parameters define how hard a note is struck and how fast or slowly the note is released. Having the data in MIDI format now allows me to digitally control the performance to produced a large number of variations on this theme. For example, by speeding up and slowing down the tempo and by using a wide range of instrumental voices to play the musical phrase, I can hear it in many different ways. By creating these variations, I can explore the underlying rhythms and structure of the music from an aural point of view.
Once this has been accomplished, I will look at the graphic elements of the music. Musical notation is by nature a visual picture of sound. One of the shortcomings of musical notation is that it evolved in a period of acoustic instrumentation. Although open to interpretation by various musicians, the palette of instruments available at the time was limited. Several new notation systems have been developed over the past several years for electronic and digital music, although no standard has emerged; these new systems are more often that not a way for individual composers to visually document their music. In my case, the original score will be translated into a 3D software package based on the number, pitch and duration of the notes. I will use the numeric data as supplied by the MIDI file. In addition, I usually sketch ideas while listening to the various interpretations I have recorded of the musical phrase.
Out of this process starts to emerge a visual picture of the music and a design for the sculpture. The music has evolved from the written score, to a MIDI performance with numerous interpretations to color sketches made while listening to the music; from there it develops into a three-dimensional database. This database allows me to visually play with the data, the same way that I played with it musically. I can apply different geometrical shapes and color to the data, and can view the data from a variety of viewpoints.... Notes from the score were translated into MIDI data, and then entered into Alias PowerAnimator software. ... I will use this same process with the data generated from the Mensaert composition, resulting in a design for a sculpture." (1)
The Creative Process
I would now like to describe, in specifics, the creative process of producing the interactive CD-ROM and sculpture of Jan Mensaert's musical piece. The first part of the process involved making several recordings of the music. The instruments we started with were piano and guitar. Once the music was recorded, I listened to it over and over until I had it memorized and could hum the melody to myself. This now gave me a much deeper understanding of the music. While hearing a piece of music once is fine from the musical appreciation standpoint, memorization puts the entire piece very clearly into my mind.There were now two things to be done. The first was to produce the CD-ROM, which revolved around the music, and the second was to conceive of a visualization of the music in a physical form.
The process of making the CD-ROM started with the recording of the music. I had decided to do several variations of the music in order to allow the listener to hear it in a variety of ways. By using different instruments, I can evoke slightly different moods and expand the appreciation of the listener by allowing them to hear the music in several different ways. MIDI made this process much easier. I started by working with Jeff McGowan, who is an accomplished keyboard player. We recorded the musical phrase both in a digital audio form and with MIDI. Once we had recorded a performance that we were happy with, I then used a Roland JV-1080 Synthesizer Module, which has several hundred voices of a wide range of musical instruments. By auditioning the voices, I then chose several which I thought matched the composition the best. The music has a distinctive classical European flair to it and those were mainly the type of instruments that were chosen. The initial group consisted of piano, pipe organ, a symphonic string sounds, an electronic flute, strings and bells. As the piece evolves, I will record additional musical voices and interpretations for the piece.
After the music was recorded, it was put into a file format that was readable by Macromedia Director, a software used to created interactive work. I then programmed the work, so that the user can click on a title an hear that particular musical interpretation. This is then transferred to a CD-ROM that can be played on a Macintosh or PC computer.
Conclusion
Digital art allows artists more creative freedom. Music can be interpreted as an image, and vice versa. My goal in creating this piece, "Variations" is to open the imagination of the viewers. Rather than hearing a single recorded interpretation of the music, they now hear several instrumental interpretations. They also see an artistic interpretation of the music in abstract imagery, and view a sculptural representation. The existence of a simple music stand containing the sheet music reminds them of the original environment the music was performed in. The Glass Bead Game invented by Hermann Hesse allows for the symbolic transformation of the original creative inspiration of a work of art, music or literary work into the various media that artists work in. This essay is one interpretation of Jan Mensaert's music. The music on the CD-ROM and the associated imagery and sculpture are others. One of the purposes of art is to change the viewer's perception. It is hoped that after the viewer experiences "Variations", they will think differently about music, image and sculpture.References
1. Bruce Wands, "Digital Salon Chair's Statement", LEONARDO, Vol. 32, Number 5, pp. 349-351, MIT Press, 19992. Hermann Hesse, Magister Ludi (the Glass Bead Game), trans. Richard and Clara Winston (New York: Bantam, 1986 (1969)), pp. 68, 178.