Note: Synestesia Software has had many changes after 2003 and it has also got a wide range of new features.
I still hope that the translation below could be of some help until the updated version of the method will be available.

Lauri Gröhn: Music from Pictures 2003

This text is based on the following presentations (in Finnish):
Producing Generative Music, University of Helsinki 24 November 2003 (pdf, 1.2 Mb)
Music from Pictures, University of Jyväskylä 11 December 2003 (pdf, 1.22 Mb)

1 Introduction

Using the method I have been developing one can generate music (midi files) from any pictures. Sculptural work is a good metaphor for the method: the sculptor starts from a block of stone or tree which inspires toward a certain end result. Synestesia Method is based on filtering pixels away and on using several selectable parameters. Going back to the sculpture metaphor, using parameters reminds of looking the sculpture from different angles or in lightning and environments. The picture on which the music piece is based and the list of parameters together form the metascore of the composition. The challenge is to find pictures generating interesting music.

On the pages one can find about 100 pieces (mp3, midi) which have been generated by using many kinds of pictures: landscapes, drawings, barcodes, Nokia's logo, European Union flag, one euro coin, cash receipt, ruins of WTC, falling space shuttle, hand writing, portraits, dogs, Aurora Borealis and so on. In a way the picture correspond the title of a "normal" composition.

Synestesia Software (Java applet) is a development system for computer aided music composition. Having no independent user interface the sofwarecan be run by using Borland's JBuilder Jave development environment. The typical configuration contains JBuilder user interface, picture file folder and midi file folder (appendix 1). When some parameters have been selected (changing the default values), the music (midi file) is generated in a few seconds (app. 2).

So far I have not done any post-processing of the midi-files. One can see from the raw score (app. 3, app. 5) that if one wants to get the music to the form used by live instrumentalist, one has to work up the score (app. 4).

The Synestesia generation will produce complete compositions, which can be (depending on the parameters) minimalistic, new age or heavily clustered music etc., the genre being a kind of "classical art music". By forming proper parameter sets one could build different styles of "composer profiles". The piecescomposed have lasted from one minute to 10 hours. The software is not using any randomness at all. There seems to be various spin-offs of Synestesia Software to many directions (app.6).

2 Filtering the Picture

After geting the picture the software adds the r/g/b values of every pixel (mode 0) or muptiplies them (mode 1). On every filtering round (parameter) those pixels having the largest pixel value are "carved away" (remember the scuplture metaphor). On image 1 the original picture is above, the next picture has been filtered (in mode 0) 10 times and 400 times. On the pictures below (mode 1) filtering has been used 100 times and 400 times. For the pixels left after filtering black colour is shown. Typically less than 10 % of the pixels (left after filtering,) are used in the following phase.

This picture shows which pixels were
used in this generation. Pixels for different
movements are marked with different
colours. The horisontal lengths of the
colored lines depend on the lenghts of
the notes. In one movement the vertical
position of each pixels depends on the
pitch, but the relative pitch of each
movement depends on the parameter
"pitch difference" described below.

3 Dividing the Picture

Depending on the number of tracks (parameter) the pictures is vertically divided into blocks.

The number of movements (parameter) defines how many pixels are available for every movement. On movement one dark pixels are used and later moments get the lighter pixels. The most common colour (black) has been filtered away). ( In the image all pixels has been shown as black.)

4 About the Parameters

Instruments, velocities etc. can be selected by using parameters or by using parameter sets where every track has its own values. The harmonies are built by selecting the number of vertical notes, minimum interval and pitch range (if smaller than the standard pitch range of the used instruments).

If the maximum pitch (parameter) has been set to 96 and the difference between the track pitch levels is 6, the highest pitch of track one is 96, track two 90 and so on. The software contains a table where the default pitch ranges of midi instruments are defined.

5 Reading Pixels

Pixels (changed to gray) are read vertically from top to down one track at a time depending on the harmony (parameter i) and thickness of the harmony (parameter m). If a pixel is missing a step i down is taken. The track is read from left to right as many times as there are movements (parameter) looking at only the pixels belonging the that movement. The time axis goes from left to right and the tempo (parameter) and the number on moments defines the lenght of the piece.


6 Pitches

The pitches depend on the highes possible pitch of the piece (parameter), the relative pitch of the track compared to one above (parameter) and the selected scale. Any new scales can easily be created.

Midi pitches and velocities are used. Every track can have its own scale if needed. The following image shows how pixels are mapped to scales.

Remark: If one selects from the original picture two pixels, it is not necessarily possible to tell which is "played" earlier and which has higher pitch.

7 Note Lenghts

A note will be triggered when on a certain track/movement one has a "rising edge" (analogy from process control) when foloowing the pixels horizontally and the previous pixel was not used (usable). The note lengths are determined with the number of usable pixels starting from the triggering one. Notes for every track have the maximum length (parameter). On image A the parameter is 7 and on image B it is 4. The unit is minimum note parameter) that can be 1/8, 1/16 or 1/32 (parameter).

8 Velocities

The velocities of all notes are defined like this:

9 Some Specific Points

Because it was set as a design parameter that all pictures should be usable for music generation some details have produced a lot of extra work. Depending on the picture it is possbile that too long pauses are generated. That's why there exist a parameter that defines the maximum pause. Too long a pause is reduced to the maximum allowed length. The start of the pieces and the end of the piece are using a special filtering so that too abrupt bursts of notes or stops of notes are elinated.

10 Last Remark

The Synestesia Java applet has at the moment about 500 lines of code. Because there is no programmed user interface, no object oriented code has been used. The code is highly modular, still. As a development environment the software has served extremely well without any need for going to back technical details. Propositions for additions given at Viitasaari Time of Music composing workshops have demanded only addition of few lines of code.

The most popular question concerning Synestesia has been the inverse process, from midi to pictures. Going back and forth in this way one could get a kind of generalized cell automaton creating music and pictures forever. Perhaps some day... (Note: those mappings are irreversible)

Appendix 1: The development environment in parameter selection phase


In the middle Java source code in JBuilder 9 environment. The generation starts by pressing the blue area on the left. The generation will take a few seconds. On the right part of the picture the folders for pictures (.jpg) and generated midi files can be found.

Appendix 2: The screen after the generation


On the bottom one can see the original picture, left to it the gray tone picture where the the filtering was started. Above right the filtered picture (gray pixels marked black) and on the left the pixels used for pitches.

Appendix 3: Raw score printed from from midi file

Appendix 4: The processed midi file

Appendix 5: Another raw score

Appendix 6: Ideas for using Synestesia Music

Generating music from mobile phone MMS messages

Tool for composers: idea generator /tester, music writer. Everybody can
be a composer, even children

Music for meditation, music therapy
Active therapy tool by making pictures and music

Tool for music education at schools, educational music, music for ability tests.

Send us a picture and get back CD (horoscope, signature, face, house, favourite landscape...)

Research tool, new way of analysing music, writing articles of this new approach

Presents: a picture or painting with CD music, CDs of the picture from which
the music came from

Tool for blind people for "looking" pictures or their drawings, listening structures

Interactive Internet service: give the
URL or draw a picture and listen to it immediately.

Generating musical illusions Association game: looking and listening, then describing what was going on. Or using a wrong picture...
Generating Ring Tones for
mobile phones from pictures

Competitions in TV: viewer send picture, the music is presented and the viewers select the best

Listening company logos
Listening the package covers

Musitives Radio Channel. Unlimited amount of background music, always different

Generating music using mobiles phones with digital camera and listening everything around

Microtonality is easy to add if needed...