What you see is what you hear: mirror neurons and music, part VI

Robert Schumann wrote in a review of a Franz Liszt concert in Dresden in 1840: “It is unlikely that any other artist, excepting only Paganini, has the power to lift, carry and deposit an Liszt_at_the_Pianoaudience in such high degree. . . In a matter of seconds we have been exposed to tenderness, daring, fragrance and madness. The instrument glows and sparkles under the hands of its master. . .It simply has to be heard – and seen. If Liszt were to play behind the scenes a considerable portion of poetry would be lost.” (Schumann on Music, trans. and ed. by Henry Pleasants)

And Marguerite Long, the great French pianist, commented about Debusssy that “he was an incomparable pianist. How could you forget the suppleness, the caress, the depth of his touch! As he glided with such a penetrating softness over the keyboard, he kept close to it and obtained from it tones of an extraordinarily expressive power. There we find the secret, the pianistic enigma of his music. There resides the special Debussy technique. . .” (Famous Pianists  & Their Technique, Reginald Gerig)

Schumann knew that he had to see Liszt perform to experience the full range of emotions that Liszt was portraying in the music. For Long, the expressiveness of Debussy’s playing was connected with his physical approach to the keyboard – which also had to be seen.

So Dr. Chia-Jung Tsay’s research, showing that visual information appears to be dominant when listening to performances (discussed in two earlier posts) shouldn’t have come as a surprise. And yet, we classical musicians still tend to think that a performance is all about the sound – about the music itself, not about what the audience is seeing.

We understand that we use movement to create sound, and different kinds of movement create different kinds of sounds, but we usually think of movement in the service of the music, not as something that has a kind of performance life of its own. Unless, of course, we’re talking about popular music, and then it’s a whole different ballgame.  Think about the provocative (at the time) performance style of Elvis and the impact he had on audiences.  Popular music was never the same again.

Body movements of a performer really do influence what an audience member hears.

Jane W. Davidson is a professional singer and psychologist who has researched and written extensively about body movement in performance since the early 1990s. Prior to her work, researchers had focused almost exclusively on sound in the study of music perception. As an opera singer, Davidson knew that body movement must play a role in the perception of a performance, so she designed an experiment in the early 1990s that would determine what information was conveyed by a musical performer when s/he was asked to play the same musical excerpt with three different kinds of expression.

She used an interesting technique called “point-light technique,” developed by Swedish
psychophysicist Gunnar Johansson to show biological motion as moving points of light. Click on this You Tube video and fast forward to about 4:25 to see a demonstration of various human motions done with this technique. You won’t have anyScreen shot 2014-07-17 at 5.05.18 PM difficulty seeing that these moving lights are, in fact, people and you will know exactly what they are doing – just from seeing a few points of light.

Today, several decades after the technique was first developed, we look at the video and our impulse is to think that it is some kind of digital animation, but this is a photographic process using real people.

Now imagine this technique being used with musicians playing an instrument. I would love to see a video of that, but haven’t found one.  It would be fascinating!

Davidson conducted two experiments. The first looked at the performances of four violinists, the second used a solo pianist. Observers all had some experience listening to and seeing music performances. Each performer chose his own music – the violinists chose Bach, Mozart, Mendelssohn and Kreisler; the pianist chose Mussorgsky. The performers were asked to play each excerpt three times in three different manners: 1) deadpan – minimizing expressive interpretation, 2) projected – as they would play in a public performance, and 3) exaggerated – overstating all of the expressive features of the music.

Performances were recorded using the point-light technique and then excerpts were prepared in three formats: visual only, sound only, and vision plus sound.   The excerpts were randomized separately for each observer, and each observer heard all of the excerpts. The observers were asked to rate the expressivity of the performances on a 7-point scale, from deadpan to exaggerated.

By now, you may not be surprised to learn that, just as with Dr. Tsay’s research, vision turned out to be more informative than sound in the observer’s understanding of the performer’s expressive intentions. Dr. Davidson’s research was done twenty years prior to Dr. Tsay’s using different techniques, but the results are very similar.  Vision appears to give the observer or audience member more direct access than sound to the expressive features of the music – even when seen as points of light.

Are mirror neurons a way to explain the results of both of these studies? Obviously, there is something about the way in which the brain processes sensory information that caused Schumann to rely on seeing Liszt to get the full effect of his musicianship, and Long to rely on how Debussy looked at the keyboard to fully appreciate the expressiveness of his performances. And the results of both Tsay’s and Davidson’s work show that the visual aspect of performance seems to convey more information about the emotion or expressiveness of the music than the auditory.

Most pieces of music involve multiple emotional states, not just one, and there obviously isn’t time during the performance for a listener to cognitively process what the various body movements of a performer may mean in terms of emotional expressiveness. Understanding the emotions of the musical work must be a far more instantaneous process, and that sounds remarkably like mirror neurons, those specialized brain cells that allow us to understand the actions of others by mirroring those actions in our own brains.

So, you may be thinking, it’s one thing for a musician to be able to mirror the actions of someone playing his own instrument.  But what about a non-musician?

Several years ago, after I had read Marco Iacoboni’s Mirroring People, I e-mailed him with several questions related to teaching and performance.  He very graciously sent lengthy responses.  In answer to one of my questions, Iacoboni pointed out that even though he is not a musician, his mirror neurons are at work when he watches a performance.  He went on to say that playing an instrument is about using body parts – hands, fingers, mouths – parts that we all use.  And the actions that musicians make are not totally dissimilar from everyday actions that we all make.  So at least in part, the non-musician (as well as the musician) is simulating movements similar to the ones used by the performers, and that is what allows him to understand the expressive nature of the music.

Next time:  more on the audience, and the wrap-up of mirror neurons


4 responses to “What you see is what you hear: mirror neurons and music, part VI”

  1. loren.amacher@facebook.com Avatar

    Indeed! But to me, the auditory impression is as important. My brother is an ‘auditory’ pianist, and in his youth, his mood determined how the same piece sounded, perhaps on successive days. And although watching Garrick Olsen play some Bach is an interesting experience, I’d much rather watch and listen to Paavali.
    Cheers! L

    1. Lois Avatar

      Hi Loren, Sorry if it sounds as though I’m minimizing the auditory. That’s not my intention – after all, there is no music without the auditory. Just wanted to discuss the fact that the visual is more important than we think.

  2. Lyle Sanford Avatar

    ” the actions that musicians make are not totally dissimilar from everyday actions that we all make. So at least in part, the non-musician (as well as the musician) is simulating movements similar to the ones used by the performers, and that is what allows him to understand the expressive nature of the music.”

    It’s wonderful seeing you say this – as a music therapist I’ve long thought this aspect of music making and the perception of it has been too little talked about as an avenue of musical/emotional connection.

    I’m sort of a Jack of all trades when it comes to instruments and genres – looking for whatever best gets the audience involved – and in my experience, playing banjo/guitar, standing up and moving/nearly dancing with the music, is, on balance, the best way to do that. Originally it was a way of “conducting” music making in small groups – but over the years have come to see it as a way of “conducting” an audience as well.

    1. Lois Avatar

      Thanks, Lyle. I’ll talk more next time about the musical/emotional connection with the audience vis a vis mirror neurons, so it’s interesting to hear about your experience.