Synestesia Software Music, Tekniikantie 12, 02150 Espoo, Finland .
Keywords: Pictures; Generative Music; Algorithmic Composing; Psychology of Music


The most common way to generate music from pictures has been the "Piano player method". The vertical positions of pixels are interpreted as pitches and time goes from left to right. The results using this simple method have not been impressive. The method presented here uses pictures to generate music in a different way.


The Synestesia method I have been developing generates music (midi files) from any pictures. Sculptural work is a good metaphor for the method: the sculptor starts from a block of stone which inspires the artist towards a certain final result. Synestesia Method is based on filtering pixels "away" and on using several selectable parameters. Using parameters is like looking the sculpture from different angles or in diffeernt lightings and environments. The picture and the list of parameters together form the metascore of the composition. The challenge is to find pictures generating interesting music. But why does this method work?

The Synestesia method will produce complete compositions, modern art music. By selecting specific parameter sets one can build quite different styles or composer profiles. The pieces composed have lasted from one minute to 10 hours. The generation itself will take about 5 seconds. Link for the full presentation of the method is shown in REFERENCES section.

Many people interested in Synestesia method have been asking about the inverse process, generating static pictures from midi files. By going back and forth in this way one could get a kind of generalized cell automaton creating music and pictures forever. That is possible because those mappings are irreversible.

Most philosophers of music think absolute music has no semantics. The noun "cat" refers to things called cats but sad music has nothing to refer to. Sad people walk slowly and sad music is slow but that is only an isomorphism or a cliché. Pictures on the other hand have semantics. The mapping from pictures to music is always more or less artificial, there is an infinite number of possibilities.


On the site one can find about 200 mp3 pieces which have been generated by using many kinds of pictures, landscapes, drawings, barcodes, cash receipt, ruins of WTC, falling space shuttle, hand writing, portraits, dogs, Aurora Borealis and so on. The main question here is this: how can Synestesia Software using Synestesia method generate so interesting music that month after month 5000-6000 files are downloaded from the Synestesia Internet site?


One might postulate the following paradigm set: absolute music, titled music, picture music, program music, songs.

Absolute music doesn't tell a story. It is more like ornaments in art. The titles give listeners a kind of context for listening the piece. Changing the name moves the listener to a different listening context and the piece may create new impressions. Many listeners erroneously think that the piece "represents" something. Even in art the name of a painting may be important. One can see the sunset over a lake only after hearing the name of Turner’s painting “Sunset over Lake”.

Pictures correspond to the titles of "normal" compositions. Generating music from pictures creates the contexts for listening. Using both a picture and a name may even magnify the effect. Because of the parameters it is possible to influence the piece to be generated in a pseudo-synesthetic way. Quite often the generated music seems to have its own character "asking" the user to go to a certain direction. The "picture music" is a new and unmapped area of music, connecting art and music. But one must always remember that pictorial representation of music is not possible because music doesn’t have semantics...

Program music is based on descriptive texts. Film music may be a kind of program music, or film "on the background" can make absolute music programmatic in the ears of film lovers. Good examples of this are Stanley Kubrick's films "Eyes Wide Shut" and "Space 2001" using extensively György Ligeti's music.


Can computers be "creative"? It is a well known fact that some of the most original moments in Beethoven's Ninth Symphony have been the result of printers' error. Also the Finnish composer Einojuhani Rautavaara has reported that he has got some of his ideas when hitting a wrong piano key during composing. So it seems that live composers may use randomness or serendipity as a creativity tool. It is a kind of a paradox that computer music based on randomness has not been so successful. The Synestesia Software does not use randomness at all, all pieces are deterministic.

Many creative composers have developed sets of algorithms for creating their compositions. Good music must move on midway between the expected and the unexpected. Synestesia music typically does something else than expected without randomness. Professional musicians may experience this negatively as if the music would be trying to fool them. However, the pieces are not based on old clichés that are so typical especially in film music. Even professional creative composers could profit from Synestesia Software as a creativity generator.


Synestesia music doesn’t use the model of tension/release/resolution but many pieces are still highly emotional. The structures and rhythms of Synestesia pieces are mainly based on the structures and colors of the pictures used for generations. Anybody can use Synestesia Software for generating music and the music will always be coherent when compositions of amateur composers may be less coherent.

According to an old myth Beethoven composed his pieces in his head before writing on paper. That myth has been proved to be false. The documents show that Beethoven wrote his pieces tens of times on paper before the final version. When using Synestesia system one starts from the chosen picture and one is able to listen to the generated piece after only 5 seconds. Some parameters can be changed and after additional 5 seconds the piece can be listened again. The process is very interactive and rewarding.

Sometimes Synestesia music may be like semi-repetitive tapestry still having inner variability. The method is based on gravity of pitches. Perhaps that is one of the secrets of the system.

Much of the enjoyment of modern art music is based on rehearing the music many times. Synestesia pieces can be perceived during first hearing.

Older art music has the feature of persistence of illusion, the same music can be enjoyed again and again. Synestesia music creates the same illusion.

It is well known that musical notation is only part of the truth and musicians playing the piece create their own version the piece. Synestesia midi files can be printed as preliminary scores and processed into full scale scores with articulation etc. Synestesia Software creates the control codes to be used by the software instruments used for playing midi files, but the notation or the sequencer programs don’t understand these articulations. On the other hand it is possible to use instruments in such a way that normal instruments can’t follow, unplayable rhythms and piano glissandos are good examples of this.


One interesting tool for research (and fun) is under development as an application of Synestesia method: Snapshot Music.What you see is a camera lens and loudspeakers on a side of a box. If the camera detects some change on the view the system will wait until the view is nearly static and then the system will generate music using up to 16 instruments and after a few seconds one can listen to the music. The piece will last about one minute and the process will start from the beginning. This tool could be used for experimental musicology in many ways and for music therapy for example.

The main problem at the moment for using Synestesia music is the price of the software instruments used for playing the midi files. Also the software instruments call for a powerful computer having a lot of ram. Synestesia Software (Java applet) itself can be used in any computer, PC or Mac. My guess is that in a year or two the situation will be quite different and the opportunity window will be widely open for the approaches like Synestesia method. Many new applications will be developed for “emotion conversions” from pictures to music – even for mobile phones.


Method description:


SAT1 - SAT2 - SAT3 - SAT4