by FLORIAN GROND
We think that reality represents itself as a fragment or even exists as such. Etymologically the mathematical notion of a fractal shares similarities with the fragment. Whilst the fragment always represents a part of the whole, some fractals, namely space filling curves (SFCs), have the capacity to rearrange in an orderly manner everything that they map in lower dimensions. The Russian philosopher and mathematician Pawel Florenski (1919) linked his insights about SFCs very early with questions of image theory drawn from the fine arts. Interestingly, Kazimir Malevich (1919), also expressed similar ideas by referring to literature. Without explicitly drawing links with mathematics, Malevich was very much aware of the necessity to break forms when reducing the dimensionality of media, and came close to the notion of fractal dust. Some decades later, similar lines of thought are found in the work of Italo Calvino, more precisely in Six Memos for the Next Millennium (1985). Like Malevich, Calvino is unaware of the mathematical foundations mentioned by Florenski, and therefore approaches the topic intuitively.
In the first sections of this essay, I use SFCs as a metaphor for understanding text-image relations. This metaphor provides a framework that comprises apparently disparate positions, like the skeptical or deconstructive use of language, and conventional stylistic devices. The ability of SFCs to map high-dimensional content onto one dimension is of particular importance for modern computer science, where key issues include the parallelization of processes and the organization of complex content for quick access in databases. I recently took advantage of the SFC structure in order to develop methods in the field of scientific sonification (Grond, 2007), which I outline in the final section. Wherever SFCs are applied, their purpose is to mediate between sequential or linear processes and complex, high-dimensional realities. I describe the afore-mentioned examples from the arts and philosophy and link them with today’s use of SFCs in computer sciences and media arts, in order to illustrate our relations to media of different dimensions, that is to say, of differently represented realities.
PAWEL
After having studied Russian icons, Pawel Florenski wrote his essay “The Reversed Perspective” (1919), one of his best-known writings outside Russia. In this essay he investigates whether or not there is a true or at least particularly valid way to grasp reality in an image. He seeks to prove that the reversed perspective is not only able to capture reality, but does so even better than the canonical central perspective. Florenski uses mathematics in an inspiring way in order to emphasize the validity of his detailed treatment of major questions in the fine arts. This is why we find SFCs in his discourse about central perspective, which was a rather unusual link for his time. In order to prepare images for formal mathematical treatment with SFCs, Florenski (1919) made the following comment with regards to color: “The colors correspond to the energy of an engraver in the process of engraving”. Color thus becomes a value between black and white, and we are left with something that can be thought of as a set of points on a two-dimensional surface that is ready to be addressed by means of mathematics. In the same way, a painter is confronted to the well-defined problem of translating the points of three- or even higher-dimensional reality onto the two-dimensional one. As a mathematician, Florenski knew the work of Cantor, Peano and Hilbert, and fruitfully connects their results with questions of image theory. In order to exemplify his mathematically inspired ideas, he had to take one step back. He focused on the problem of mapping the two-dimensional plane onto a one-dimensional line. Florenski took this problem as an analogy for all other possible correlations between realities of different dimensionalities. The fact that the medium of depiction is always smaller than that which is depicted, at least as far as the dimensionality is concerned, dominates his considerations about the relation between reality and the medium. Yet Florenski is convinced that it is still possible to depict everything, and that nothing from the whole will get lost. The transformation from one reality/medium to the next leaves its traces, according to Florenski, and he explains this by means of Hilbert’s curve.
DAVID
In 1890 Giuseppe Peano invented SFCs, which were of purely mathematical and academic interest in the beginning. It is often mentioned that his approach was entirely analytic, and that there were no accompanying drawings (Peano, 1890). One year later, David Hilbert (1891), to whom Florenski often refers in his work, found a particularly instructive example of an SFC, on which we focus here. Florenski never went into detail regarding how his thoughts concerning the problem of depiction and mapping are mathematically formulated. This would have been unreasonable for the readership of his time. But today we are more familiar with fractal structures. So, how can we construct a correspondence between two- and one-dimensions via Hilbert’s curve? Let us imagine a white page. This page can later be replaced with any image. On this page we start with a line segment like this Π. In the next step we construct four copies and rearrange them applying some rotations. Then, we close the gaps in between and take the resulting structure as a starting point for the next step (Figure 1).
Figure 1: The first steps in the construction of the Hilbert curve
This can be repeated at will. Every time the result has to be scaled down four times in order to fit the size of the page with which we started. For all finite approximations, this line is self-avoiding, which means that the line covers every point on the surface of the page only once. Further, neighboring relations of a 2D image tend to be maintained on the 1D line. So why do we have to undergo such an involved construction? Could we not have used a simple spiral that covers the two-dimensional plane instead? We are about to discover the traces that are left after transforming content into a medium of fewer dimensions. Although we have managed to transform everything, and nothing is lost, we have however destroyed the relations that the image contains. The middle of a spiral, for instance, preserves the relations found on an image relatively well. But as we progress outwards, the points that are close to each other on the plane are unlikely to be found as neighbors on the line, unless they are beaded in succession by chance. Even the meander-like structure of Hilbert’s curve is not able to perfectly maintain the relations in the image, yet it does so much better than the spiral. Florenski (1919) is well-aware of this fact when he describes in very plastic terms the essence of a fractal, or how the medium represents reality as broken into dust: “An eggshell, or even just a piece of it, can in no way be spread onto a marmoreal table without being deformed, crushed into fine dust, and this is why it is impossible to precisely depict an egg on a piece of paper or canvas”.
KAZIMIR
A contemporary of Florenski, Kazimir Malevich was also aware of disturbing proximity relations when mapping reality. He found a similar problem in text production. In 1919, he wrote: “When we look at a line of verse, it is minced like in a sausage, made of all possible forms alien to each other and not knowing their next-door neighbor” (Malevich, 1919). In poetry, reality has to be turned into a linear string, in contrast to its existing multi-dimensionality. If we go along this string and try to consume it, we experience the forms of reality as rhythm. According to Malevich (1919), the poet is “rearranging the repository of all things.” Here he meets precisely with Florenski´s idea that everything remains preserved, but that the rearrangement leaves noticeable traces behind. The difference before and after the transformation is not manifested in the fact that something is missing. In the sausage and in the repository metaphors, both forms stay more or less intact. Everything seems slightly distorted and mixed. Elsewhere, however, Malevich breaks form on all levels, like the eggshell spread onto the marmoreal table. He realizes that the power of rhythm, which is finally the constructive principle of mapping, sometimes leaves nothing behind untouched: “There is poetry in which pure rhythm and tempo remain as movement and time; (…) yet at times a letter is not able to embody the tension of a sound, and therefore has to pulverise” (Malevich, 1919). And in a more drastic formulation: “There is poetry in which the poet has to destroy objects for the sake of rhythm, leaving behind torn frazzles of unexpected compositions of forms” (Malevich, 1919). In connecting the mapping problem to the fine arts, as Florenski did with Hilbert’s curve, and Malevich with text production, we are guided back to the beginning of our considerations: the points in images. When we read a text, an image appears in our mind. Only by constructing this image, which should be understood as an abstract, higher-dimensional reality, can we make meaning emerge by bringing together these single pieces of pulverized letters and words, senseless on their own. The installation “hilbert01” (Figure 2) attempts to visualize this metaphor.
Figure 2: hilbert01 computer graphic installation by Florian Grond 2004
Malevich comes to comparable conclusions: “Rhythm and tempo create and take away the sounds that are born through them, and generate a new image from nothing” (Malevich, 1919). Since we generally do not know the construction principle by which our fractal line perambulates reality, we are, with reason, skeptical towards the creation of images out of words. Only the meaningless, small pieces of fractal dust seem to be real.
ITALO
In the chapter entitled “Quickness” in Six Memos for the Next Millennium, Calvino (1985) refers to “lu cuntu nun metti temptu” which means that time takes no time in a fairytale. This is a formula traditionally used by Italian storytellers to indicate large leaps in time. With this formula, Calvino investigates the relationship between text and image sixty years after Florenski and Malevich. One might suggest that all three of them were facing the construct of Hilbert’s curve, although not all of them knew it. In order to shorten time, Calvino says, the poet can iterate less deeply in areas where his imagination of reality is more monotonous. He can leave out parts that he does not want to describe in detail so that the reader can consume the text faster. In fairytales, people often disappear into parallel worlds, and when they return they find themselves again in a different time. The parallel worlds correspond to the levels of the hierarchical order of the Hilbert curve, which, in order to perambulate it, requires a different amount of time depending on which iteration depth one chooses. With the aid of Calvino we can also shed light on the essence of rhythm in prose. Calvino explains: “Just as rhyme creates rhythm in poems and songs, so can we find events that rhyme in prose” (Calvino, 1985). And he assumes that part of the childlike joy in listening to fairytales is the repetitive anticipation of certain structures, be they situations, colloquialisms or flowery phrases. If we refer to the Hilbert curve, we find that the curved structure leads the reader through the imagined in such a way as to come back to the same neighborhood several times before proceeding onwards. According to Calvino we can often find this kind of self-similarity in Middle-Eastern stories, which strongly depend upon the structure of a story within a story (Calvino, 1985). Scheherazade can save her life because she can endlessly find links from one story to another. In a way, she is being guided through reality along a fractal path, which corresponds to a further topos that Calvino describes as being important for good literature. This is the idea of “festina lente” (more haste, less speed), which is sometimes depicted as a crab and a butterfly. The butterfly stands for an agitated movement and the crab for inertia. A good text must incorporate both qualities: dwell upon one subject and browse it subtly, all the while being full of flighty movement. It must not be boring, but metaphorically and indirectly shed light on the subject, rather than come to the point and insist on it. The image that appears to us when a fractal line guides us through reality is similar. But what is the relation of possible time contraction and rhythm with the image? A good text should enable us to experience certain images instantaneously even when we have to struggle through it. In order to explain this Calvino (1985) uses the figure of Sagredo in the Dialogue about the Two World Pictures by Galileo Galilee. He describes Sagredo as someone capable of instant reasoning. This means precisely to catch sight of the image while reading a text, and to perceive what we are reading from the next higher perspective, which, in a certain way, is equivalent to taking a divine, timeless position. The next higher dimension enables us to avoid the process of piecing together an image, and therefore stops time. This is how the image confronts us with the utopia of instantaneous, and therefore timeless, insight. A good text should provoke the same experience, in spite of the fact that we spend time with it.
Florenski particularly emphasizes that we must always accept a compromise when depicting reality in media. On the one hand, we can try to maintain the original form in its depiction. This is considered a naturalistic approach. In this case, the relation between reality and its representation is not unique. On the other hand, we can try to achieve a unique relation between all points in the depiction and reality. In this case, the representation seems formless and appears to be broken up into dust. These juxtapositions of representing and conserving form versus its deconstruction seem to mutually condition each other. Breaking things up into dust might be the result of a mapping through an elaborate structure like an SFC. In my opinion, creative processes always negotiate between these poles. While Malevich for instance emphasizes the deconstructive aspects of literary creation (torn frazzles of unexpected compositions, Malevich (1919)), Scheherazade is a good example of the constructive efforts in storytelling, which we have compared to the self-similar structures of SFCs. If we think of text as a sheer information carrier, which requires a proper interface in order to communicate its content, then we can compare it to the experimental indexing of databases with the help of SFCs. The way in which Scheherazade tells her story can therefore be seen as an early realization of a database, in which the form makes content highly accessible. Texts can only contain a finite amount of signs, which are only repeated a limited amount of times. Still, our mind fills the rest to infinity, even when the text only allows for a finite repetition of self-similar structures. A thousand and one nights is a finite amount of time, yet Scheherazade becomes eternal by telling her story. What is particularly charming about a meandering narration that is self-similar in a mathematical sense is that through its very structure, it has the power to refer beyond itself and its finite representation.
SFCs in sonification
All these ideas about SFCs recently inspired me to apply them in the field of sonification (Grond, 2007), similar approaches can be found in (Vogt et all 2007). As an example of high-dimensional data I took sequences of images, as in a movie. The movie frames, which were reorganized into lines through SFCs, were prepared for sound synthesis. Considering the fact that an image transmits information in a synchronous way, I took the linearized data as frequency information in order to achieve a sonification with similar qualities. The relations of frequency ranges in the sound reflected the relations within the whole image. The sequential RGB (red green blue) output of the image data was scaled between 0 and 1, and further read into three different audio buffers of 1024 samples. These buffers were used in subtractive audio synthesis as a filter bank for white noise. The three different resulting audio streams were sent to a pair of stereo channels, with one stream equally distributed to both of them. Comparing different data scans, as well as their effects on the resulting sounds (Figure 3), I found that a horizontal scan line creates repetitive patterns of shapes in the image.
Figure 3. This image shows different scan methods applied to a movie file of 85 frames. The vertical axes are the scan points and the horizontal axes are the 85 movie frames. From top to bottom: spiral scan, line scan and SFC scan (Hilbert curve)
If we look at them from a sonic perspective we recognize something like an equally distanced overtone series, which has a very pronounced effect on our acoustic perception. This fact leads us to conclude that some parts of what we hear in this case are structural artifacts of the scan process. If we take a spiral instead, the overtone patterns are more complex, but are essentially still there. By using an SFC, we avoid such patterns, thanks to the locally progressing nature of this scan process. In order to use the program to explore movies, I implemented methods to interact with it in real-time during sound synthesis. The acoustic structures that the user might hear would be repeating oscillations in a certain frequency range. In this case, the program allowed to restrict scanning to this frequency range, and to mute all other frequencies. The segment of the scan line that represented this particular range was displayed as a line textured with the scanned image. In this way, a correspondence between acoustically interesting frequency regions and the relating parts of the image could easily be established. Without SFCs it would be impossible to narrow down the corresponding frequency range. This usage of SFCs basically corresponds to range queries in databases as described by Zimmermann (2001).
Using the SFC took advantage of the close relation between the image data and the sounds produced, so that any perceived sound pattern could be well identified to its visual cause. The biggest strength of SFCs in sonification is to open up the possibility that perceived acoustic patterns might point directly to specific features in complex, high-dimensional data. In this way, the SFC proves its validity once again as a mapping method, by providing an efficient interface with which to access high-dimensional structures.
Acknowledgment:
I would like to thank Tamar Tembeck for her revision of my English translation of a first version of this article in German (Grond, 2005). Agnes Grond, Gunther Reisinger and Inge Hinterwaldner made numerous fruitful comments to the original German essay. I would like to thank Julia Lechler for pointing me to Pawel Florenski and Kazimir Malevic.
Calvino,I (1985) Six Memos fort he next Millennium, Harvard University Press, Cambridge, Massachusetts 1988.
Florenski, P. (1919): Die umgekehrte Perspektive in Raum und Zeit, KONTEXTverlag Berlin 1997
Grond, F. (2005) Von der Realität zur Linie und zurück, eine kleine theory of everything, In Das Wahre, Falsche, Schöne. . Grond W Mazenauer B (Eds) Studienverlag/Haymonverlag, Innsbruck 2005.
Grond, F. (2007) ORGANIZED DATA FOR ORGANIZED SOUND Space filling curves in sonification Proceedings of the 13th International Conference on Auditory Display, Montréal, Canada.
Vogt, K de Campo, A Frauenberger, C Plessas, W Eckel, G (2007) Sonification of Spin Models. Listening to Phase Transitions in the Ising and Potts Model Proceedings of the 13th International Conference on Auditory Display, Montréal, Canada.
Hilbert, D (1891): Über die stetige Abbildung einer Linie auf einem Flächenstück, Mathematische Annalen 38
Malevič, K (1919): Über Dichtung, in: Kazimir Malevič, Gott ist nicht gestürzt, Carl Hanser Verlag München Wien 2004.
Peano, G (1890) Sur une courbe, qui remplit toute une aire plane Mathematische Annalen.36
Zimmermann, J (2001), Dynamische Lastverteilung bei Finite-Elemente-Methoden auf Parallelrechnern mit Hilfe von space-filling curves, Technical University of Munich, Retrieved September, 2007, from:
http://www.cip.informatik.uni-muenchen.de/%7Ezimmermc/sfc/zula/
MutaMorphosis: Challenging Arts and Sciences 