Seeing sounds, hearing colors, part II

Imagine if you saw a color whenever you looked at someone’s face, and different faces were different colors.  Or tasted eggs when you heard the word “fax.”  Or saw a mental map placing any number you saw or heard in a certain location in space (as in Number formthe image at the left, called a number form).  Or had a bitter taste in your mouth when you heard a major second and a salty taste when you heard a minor third. These are all forms of synesthesia, the involuntary physical experience in which a stiumulus to one sense automatically triggers a sensation in a second or even third sense.  

In the last post, I wrote about research that shows that we all have music – color associations, but not to the degree experienced by someone who has synesthesia.  So in the next two or three posts, we’ll look at what synesthesia is, at some of its history, at what is actually happening in the brain that causes it to occur, at sound-color synesthesia (the kind that, as musicians, we’re probably most interested in), and at the connection between absolute pitch and synesthesia (who knew!).  And does it actually make a difference to a musician if you have synesthesia or not?

Scientists have known about synesthesia since the late nineteenth century.   In 1871, the German psychologist, Gustav Fechner, wrote about a survey he had conducted that found 73 people reported seeing colored letters.  The English polymath Francis Galton was also working on this unusual combination of the senses, and he published Inquiries into Human Faculty and Its Development in 1883, in which he relates several accounts of individuals who saw colors when seeing letters, dates, or months of the year.  Soon after Galton’s publication, the term synesthesia was coined based on the Greek syn (together) and aesthesis (perception).  So synesthesia literally means “joined perception.”  But Fechner’s and Galton’s ideas didn’t get much traction.

Scientists throughout the twentieth century have scoffed at the idea of synesthesia or dismissed it as fakery.  According to researcher Richard Cytowic, scientists “rolled their eyes” whenever the subject came up, thinking that people who claimed to have synesthesia probably just had strong memories from childhood associations, such as that of colors with letters (the most common form of synesthesia), perhaps from a picture book in which letters were shown in different colors.

There was strong interest in synesthesia between the 1880s and 1920s among artists and musicians. This fascination was reflected in an exhibit titled “Visual Music” co-organized in 2005 by The Hirshhorn Museum in Washington, D.C. and The Museum of Contemporary Art, Los Angeles.  I still remember entering the exhibit at the Hirshhorn with the splash of music and colors and the eye-opening way in which the curators showed an alternate view of the abstract art of the twentieth century by looking at the works of artists who were exploring ideas related to synesthesia.

For example, Wassily Kandinsky, who was a synesthete, “used color to associate tone with Kandinsky Fuguetimbre, hue with pitch, and saturation with the volume of sound, as seen in his groundbreaking painting Fugue (1914)” (from the MOCA press release).  (Fugue seen on the right.) Georgia O’Keeffe, not a synesthete but passionate about music as well as art, explored the idea that music could be translated into something one could see in her Music, Pink and Blue No.2  (1918) (seen below left).

o_keeffe_musicpinkandblueno.2_600I could go on about the art and music connections explored in this exhibit, such as color organs, videos, and light projections, but I digress.  Suffice it to say that musicians and artists early in the 20th century were extremely interested in synesthesia and what it meant in terms of making art.

But from the 1930s to the 1980s, interest in synesthesia dried up – definitely in the scientific community (not much there to begin with) and to some extent among artists and musicians.

Then in the 1980s, a neurologist named Richard Cytowic began exploring synesthesia and interest was rekindled both in the scientific community and among the general public.  You may recognize his name as the author of the book The Man Who Tasted Shapes.  While he deserves credit for bringing synesthesia to the public’s attention, research in the 1990s and 2000s has led neuroscientists to different conclusions than those drawn by Cytowic in his early work.  And of course, with neuroimaging, researchers can now see the effects of synesthesia in the brain.

In 1999, Vilayanaur Ramachandran and Edward Hubbard at the University of California at San Diego began an extensive study of synesthesia.  One of the problems all along had been the fact that there had been no objective tests to determine synesthesia; it was all self-reporting, and there was no way to tell if someone was actually seeing colored letters, for example, or if they just had a strong mental association.  So Ramachandran and Hubbard developed several tests.  My favorite is something called a pop-out.  When non-synesthetes look at the black fives and twos on the white background shown below on the left, we can find the twos, but with a bit of work.  A synesthete with grapheme-color synesthesia will look at the square and the twos will “pop-out,” because the fives and twos will be in two different colors, as in the reds and greens shown on the right.  Other kinds of tests were devised to determine other kinds of synesthesia.

Ramachandran pop-outSo by the early 2000s, with the use of Ramachandran’s and Hubbard’s various tests, there was conclusive proof that synesthesia does, in fact, exist.  So the next question was, why does it exist?  What is happening in the brain that causes an input to one sense to trigger another?  What is happening in your brain if different musical pitches or keys are different colors for you?  And does it matter as a musician?  More on that in the next post.

In the meantime, you may enjoy watching this TED talk by Neil Harbisson.  Harbisson was born with total color blindness and sees only in grayscale.  But a new device he wears turns color into sound frequencies, so he actually hears color as music, the ultimate synesthesia experience.  He wasn’t born a synesthete, but technology has made him one and has allowed him to experience color through music.


6 responses to “Seeing sounds, hearing colors, part II”

  1. Loren Amacher Avatar
    Loren Amacher

    Great post, Lois. Fascinating topic, and waiting for more!

    1. Lois Avatar

      Thanks, Loren!

  2. Kyle Gann Avatar
    Kyle Gann

    Hi Lois! Catching up on your blog, which is very well researched and elegantly written. I was interested to learn that color-letter association is the most common type of synesthesia. I remember as a young child being convinced that the letter B was navy blue, E yellow, F light blue or bluish-gray, D red, A green, M black, N brown, Q pink, and so on. I still have these associations today, and I don’t think they came out of a child’s alphabet book.

    1. Lois Avatar

      Hi Kyle, “elegantly written” from such an accomplished writer as you is high praise. Thanks so much! I had no idea you are a synesthete, but you’re right. Those probably aren’t associations from an alphabet book. They are too specific and consistent over time. I’m curious – when you are composing – do you ever think of those colors in connection with the letters of the musical alphabet?

      1. Kyle Gann Avatar
        Kyle Gann

        I think of the key of D as red, and E major as yellow and E-flat as kind of gold, but it has nothing to do with the sound, just the color of the letters. I do think the nuance of just intonation intervals (different major third sizes, for instance) offers an unexplored analogue to color perception. And yes, I can see you spend a lot of time polishing your posts.

        1. Lois Avatar

          There is recent research showing a relationship between perfect pitch and musical key – color synesthesia. If you see the key of D as red and you are used to standard tuning, there will definitely be a conflict of perception if you are hearing something in the key of D played in Baroque tuning – can be incredibly upsetting. I actually thought about just intonation as I was writing about music-color synesthesia and wondered about the effect – would be a great area of research. But who would think of it other than a composer who works with – or a performer who plays – music in just intonation.