Are you listening or watching? part II

SimonI hate to admit it, but I played the electronic game Simon for years before I realized it was possible to play it by the colors, not by the pitches.  I knew the red, blue, yellow, and green buttons each corresponded to a certain pitch and flashed when that pitch sounded, but it simply never occurred to me to remember the sequences by color.  For me, the game wasn’t visual, it was auditory – I memorized the pitch sequences.  I still remember my amazement when I learned that it could be played by color, not sound.

I felt somewhat the same jolt to my brain after reading Chia-Jung Tsay’s research that I wrote about in the last post.  Quick recap in case you haven’t read it yet:  through a series of experiments involving 1164 participants (I incorrectly placed the number at 900), she demonstrated that both novices and professionals, in spite of saying that auditory is the most important component in judging solo musical performances, were far more accurate in identifying winners of competitions when they had visual excerpts only, as opposed to auditory excerpts only or excerpts with both visual and the music.

Of course, learning that vision may take precedence over auditory in our brains when listening to live performance is a far greater shock than learning, all those years ago, that people played Simon by the colors.  After all, I’ve spent my career as a performing pianist and teacher, and I would like to think that people are actually paying attention to the sound of the music when I perform.

Don’t get me wrong.  I know how important the visual is in performance.  In fact, some of my favorite brain research involves mirror neurons, those specialized brain cells that fire not only when we perform an action, but also when we see someone else performing that action.  Mirror neurons do impact music performance, and we are going to begin talking about them in my next post.   But the real mind bender about Tsay’s research is the fact that the participants who scored well above chance in identifying the winners of the competitions were those who had visual information only.  It wasn’t visual combined with the music.  It was visual all by itself.

When learning of Tsay’s research, singers say – of course.  We know how important the visual is because we’re trained to act out the music.  We have a text that is about something, and of course when we’re singing opera, we’re acting.  Even when singing lieder, we use body movement and facial expressions to help convey the meaning of the song.

And a friend who is a long-time presenter of concerts reminds me of performers that neither one of us has responded to enthusiastically when hearing them in recordings, but whose live performances we have really enjoyed because their way of playing, their body language, has reinforced or illuminated what we were hearing.

On the other hand, you have only to read a review such as this one, by New York Times critic Anthony Tommasini, to see what a negative impact the visual can have in a live performance.

And lest anyone think this research applies only to individual performers, Tsay has published a second paper dealing with research conducted with chamber ensembles and orchestras.  In one experiment, novices were given excerpts of finalists from international chamber music competitions in the same three formats as in her earlier study:  sound only, video plus sound, and video only.  As in the earlier study, participants who had the sound only excerpts identified the winners (as chosen earlier by the competition judges) at a rate below chance.  Those with video only identified winners at a rate above chance.  Participants with the video excerpts listed the following reasons for their choices:  visual communication, expression, movement, passion, involvement, eye contact.

Tsay commented in an e-mail that “gestures can reflect the physicality that produces certain touches, tones, and colors that underlie expressive sound.  There is something about visual information that is better able to convey cues such as passion or involvement or creativity.”

silhouette of quartet

Another experiment gave paired excerpts from two orchestras to non-musician participants.  Again there were three kinds of excerpts:  sound only, video only, and sound plus video.   One of the two orchestras in each pair was a top-ranked professional orchestra (including the Berlin Philharmonic, Vienna Philharmonic, London Symphony, Chicago Symphony, and several others), and the other was a regional or university orchestra.  Musical content was identical in each pair of excerpts, and the orchestras were shown in similar kinds of halls, wearing similar concert attire.  The conductor was not visible in the video clips.  While it would seem that there shouldn’t be any question in choosing which was a professional orchestra and which was a regional or university orchestra, those participants in the study with visual only excerpts selected the top orchestra a higher percentage of the time than did those participants with the sound only excerpts.  So visual information was still a better predictor.

One of the reasons I am so fascinated with brain and music research is that the best of it really challenges my thinking about performing or teaching.  Sometimes there is an “Aha!” when I see scientific confirmation for something we musicians have been doing all along.  Other times, the research is a real eye opener – like Dr. Tsay’s research.  While her research results may be initially disturbing to musicians (and Tsay says the results were initially disturbing to her as well), the idea of the importance of the visual really begins to resonate as one thinks about it.

Picture in your mind a performance that you have heard recently.  You no doubt remember how the performers looked, not just how they sounded.  In fact, it may be easier to call to mind a visual image than an auditory image.  Were you moved by the performance, or turned off?  How much did the visual have to do with that?

In both papers, Tsay mentions the possibility of emotional contagion, which is the tendency to synchronize emotions with those around you.  For example. if you are around depressed people all the time, it is more likely that you too will become depressed.  She suggests that perhaps we unconsciously come to share the emotions exhibited visually through a performer’s body language when we are listening to live performances.

The neurological basis of emotional contagion, according to recent research, is mirror neurons, which I mentioned earlier.   Neuroscientists have found that musicians have both an auditory and a visual mirror neuron system and the implications for performing (and teaching) are fascinating.   So next time:  mirror neurons.

In the meantime, if you go to a concert, think about the visual information you are receiving and what the impact is vis a vis the sound.

2 responses to “Are you listening or watching? part II”

  1. Lyle Sanford Avatar

    I’ve long felt that music “touches” us because it is, in part , physical gestures made audible – that there’s an emotional analog to all those articulations and phrases. When string players caress their instrument, we feel the caress. And with mirror neurons we can have the feeling of doing the caressing ourselves. Evolution has given us tremendous powers of discrimination when it comes to interpreting the gestures of others. That seeing physical gestures gives us more reliable info than just the aural analog of gestures makes sense as long as we’re talking about the general public.

    My intuition is that what’s you’re talking about here is hugely important. For me it’s one of those things that once you see it you wonder why it took so long to realize.

    1. Lois Avatar

      Thanks, Lyle. As you say, it is hugely important, and we’ll be talking about it for awhile. More to come.