EXTREME INTERVALS AND SENSORY FUSIONS: THE HINGE

By SHER DORUFF

“The synesthetic hypersurface refracts the activity of matter […] It is the hinge-plane not only between senses, tenses, and dimensions of space time, but between matter and mindedness: the involuntary and the elicited.” (Massumi, 2002)

It can be argued that extreme cognition is always already. Its possibilities and limits expressed in the half-second interval between brain stimulus and conscious perception. In that interstice, cognition, hallucination, memory and perception are indistinguishable as potentiality. Physiologist Benjamin Libet called this durational gap readiness potential (RP). Here, matter meets mindedness and past meets future in the illusory “backwards referral” of conscious experience; neural activity prepares a movement before the decision to move is made. Synesthesia research, the study of cross-modal correspondence and the RP interval together provide a resonant playing field for creatively thinking through the dynamics of thought as it melds with sensory experience of the world. One of synesthesia’s primary conditions – involuntary and elicited cross-modal perception – can be considered a link to unraveling the mysterious processing of “mind time.” Brian Massumi’s concept of the biogram exemplifies one effort to interweave these conditions by speculating on the implications of 1) the recursive duration of lived experience manifested in the half-second interval and 2) cross-modal connectedness.

Synesthesia discourse phase shifts through transdisciplinary vectors of the arts, sciences and humanities, tending to pin an unfortunate focus on the artistic practice of oft-touted ‘synesthetes’ such as Scriabin, Kandinsky, Nabakov, Rimbaud, Baudelaire and Hockney (Harrison, 2001). This ensemble of usual suspects, predominantly white, male practitioners with cross-modal referencing tendencies, generally depicts the practice of aesthetic, metaphorical referencing between sound, color, letters and shapes. Transdisciplinary debate ebbs and peaks as to whether or not synesthetic abilities are a normative capacity for connecting modal similarities or a neural anomaly effecting a few. Artistic practice enters the discussion by way of quasi-concrete examples with an aesthetic edge. Perhaps artists simply tend to be more proficient at sensory correlation, and more acknowledged in their attempts, tainting the statistical curve. In this respect, digital media processing and cross-modal synthesis in contemporary art can be easily examined relative to these concepts, augmenting the 19^th and 20^th century allusions to painting and music.

SYNESTHESIA: SOME ASSERTIONS

There is little consensus among researchers on the qualities and conditions of synesthesia. In the Wikipedia definition, synesthesia “is a neurologically based phenomenon in which stimulation of one sensory or cognitive pathway leads to automatic, involuntary experiences in a second sensory or cognitive pathway.” Alternatively, it is “the elicitation of perceptual experiences in the absence of the normal sensory stimulation” (Ward et al, 2006) and provides a means of distinguishing conscious perception from preconscious neural events. The two terms “involuntary” and “elicited” are critical and irreducible to any discussion of synesthesia. The American Synesthesia Association prefers to classify two overlapping forms, “proper” (clinical) and “cognitive” (addition of a sensory factor to culture sets)ⁱ. Other research terminologies distinguish between sensory and conceptual forms (Ramachandran and Hubbard, 2001). From an arts perspective, synesthesia has been called an “aesthetic appropriation of the neurological condition in which stimulation of one sensory modality triggers involuntary sensation in another” (Cox, 2005) and an “intersensory association” (Galeyev, 1999). Dissensus backgrounds this complex, provocative area of transdisciplinary research. For present purposes, synesthesia will be accorded three distinctively overlapping and no doubt insufficient categories: clinical (proper;)ⁱⁱ, physiological (everyday cross-sensory interaction) and metaphorical (intersensory translationⁱⁱⁱ).

Clinical synesthesia produces specific sensory modalities that are ‘cross-wired’ in the brain resulting in modal fusions such that one, most typically, associates a consistent and memorable color with specific letters and numbers (grapheme-color synesthesia). Research by neurologists, physiologists and psychologists reveals that the majority of all clinical synesthetes are in fact, left-handed women with color-hearing and color-letter vision and that synesthesia may be normative in all infants, receding as the brain develops (Cytowic, 1995). Physiological synesthetic conditions are shared by us all in our daily, multisensory experience of the world. The experience of food intake is an example, cross-referencing smell, taste, vision, hearing, touch and proprioception. Metaphorical synesthesia is common in varying degrees of breadth and intensity. It is a language based association and/or translation of sensory experience and has numerous varieties that overlap with the sensory couplings of clinical synesthetes. Synesthesia researchers Ramachandran and Hubbard posit that “language evolved through co-opting and finding novel uses for multiple mechanisms evolved originally for very different functions and by a fortuitous synergistic bootstrapping between these functions” (2001). Metaphorical couplings are also relegated to artistic invention. Richard E. Cytowic, a neurophysiologist and synesthesia researcher is particularly dismissive of labeling artistically elicited uses of metaphor and sensory fusion as synesthetic. True synesthesia’s phenomenology, he states, “clearly distinguishes it from metaphor, literary tropes, sound symbolism, and deliberate artistic contrivances that sometimes employ the term “synesthesia” to describe their multisensory joinings” (1995). Yet this interpretation is countered by others such as Bulat Galeyev, who embrace the metaphorical: “Synesthesia, at least of that kind which is used in common language and art, is not a “co-sensation” but rather a “co-imagination” or “co-feeling.” By its psychological nature it is “association,” specifically “intersensory association” (Galeyev, 1999).

READINESS POTENTIAL

In the 1970’s, neurophysiologist Benjamin Libet discovered through experimentation with electrical impulses on the human cortex, that an external electrical stimulation requires a 500 to 550 millisecond duration to trigger a conscious experience (Libet et al, 1979). He concluded that the conscious mind backdates the event, as an illusion, so the experience feels to be in temporally synchronus. Libet and colleagues updated this experiment in 1983 using an EEG to include an act of choice on the part of the experimental subject. The idea was to formulate the timing of the decision to move and the actual time of the movement. The subjects’ movements came 200 milliseconds after the decision but brain activity measured the activity as occurring 300 milliseconds before the decision was made. In the 300 millisecond interval of readiness potential, the brain gets a jump on conscious perception. The additional ±200 milliseconds from perception to sensation/action extends that gulf.

What might inhabit the space-time, or ‘mind time’ of these intervals? What kinds of interaction fill the temporal delay? Neural preparations for movement occur well before conscious agency throwing the exigency of the “cause” in the cause-effect equation into doubt. This all becomes exponentially more complicated as brain stimuli are not discretely linear and successive. There are multiple, most likely infinite, stimuli in any metric moment. Libet additionally discovered that any incoming stimulus modulates others before it has been actualized as a conscious sensation. Massumi describes this: “If a later stimulus can modulate an earlier one before it becomes what it will have been, the recursive durations start to meld together. Experience smudges” (2002, 196; emphasis added). That half-second is not an empty void but rather an overfull duration of multiplicities, contingencies and potentials that virtually modulate the actualizing occasions. These micro-modulations are akin to what Alfred North Whitehead’s calls “negative prehensions^iv” (1969).

THE BIOGRAMMATIC HINGE

Synesthesia’s complexity opens a playground for creative art/science/philosophy intersects. An ontological examination of synesthesia, rather than a strictly scientific methodological approach, allows a certain freedom of movement around and through controversial data and debate to explore subjective forms of perceived stimuli. Physiological sensory mashups, congruencies and feed forward spirals are of particular interest when the literal limits of quantitative scientific ‘fact’ are poached and qualified in conceptual expression. Massumi has explored this terrain in his thoughts on the biogram. He pushes synesthetic speculation to its extreme by placing perception and practice in superposition, describing the biogram as a lived, one-sided topological^v surface, a continuous and multiplicitous transformation; a lived and relived event-dimension which:

For all perceivers [the biogram] is the mode of being of the intrasensory hinge dimension. Its strange one- sided topology is the general plane of cross-reference not only for sights, sounds, touches, tastes, smells, and proprioceptions, but also for numbers, letters, words, even units of grammar. On that plane, the learned forms that are usually thought of as restricted to a “higher” cultural plane re-become perceptions. Practice becomes perception. (p.188-89)

As stated earlier, Ramachandran and Hubbard have postulated that cross-modal correspondences represent the origins of language and metaphor, of learned “higher” forms. Massumi advances that thread while shaking the fruit from the cognitive tree it’s predicated upon when describing ”experience” itself:

The cognitive model has it that “higher” forms are associative compounds built up from smaller sights and sounds as from elementary building blocks. But the workings of synesthetic biograms show that the higher forms feed back to the “lower” perceptual level. They enter the general dissolve, on a level with the elementary, fused into the surface, interwoven components of the fabric of life […] Every experience takes place in the already taken place of higher and lower, where they join for the future. Every experience is a portentous dèjá-vu at a hinge. (p.189)

Following this thought-thread there is no experiential distinction possible between the perceptual and conceptual, higher and lower forms. The “dèjá-vu at a hinge” is key, exhibiting the intrasensory dimension of the synesthetic biogram modulated in the extremely excessive RP interval of recursive experience. The non-linear past becoming future-past in every conceivable moment. It is the modulatory interaction of involuntary brain activity (matter) and elicited conscious perception (mindedness) in the duration of the half-second interval, the affects of lived experience feeding back on itself to re-potentialize, that excite fresh interest in synesthesia as a vector for pursuing the experience of perception. It’s a fascinating landscape for artist/researchers on the periphery of hard science, providing an impetus for creative lines of flight between perceiver and world.

Cross-modal congruency, a complementary research field to synesthesia, explores proprioceptive^vi relations to sensory stimuli: what is seen and what is felt in the immediacy of peri-personal bodyspace, the space defined by the reach of our limbs, the extensible border between personal space and public space. Tool use, for instance, may be incorporated into a bodily, corporeal schema or, on the other hand, the brain may extend its representation of the body into the external space of the tool. Indeed, the neural systems that represent the body itself may be difficult to separate from those that represent the space around the body (Holmes and Spence, 2004). Spatial-sequence or number-space synesthesia (see Figures 1) enters this discourse with a curious validation of a biogrammatic loop that feeds the experience of the exo-referential folding into the self-referential. Spatial-sequence synesthetes see and or sense numbers, days of the week, months of the year, floating in specific locations and can point to them in space. “There is February.” “Over there is 12.” This effect, this lived experience, involuntarily elicited, is a mnemonic device, adding another dimension of referencing. It maintains a consistent correspondence. “12” is always in the same location (and with color-letter synesthesia, always the same hue). It elicits from an informal sensory dimension between proprioception and vision through the spatial extension of words and numbers that enter the peri-personal (and sometimes the extra-personal) space of the body. It exemplifies the one-sided topological figure of the biogram. Cross-modal correspondence between sight and propriocepion, sight and touch, hearing and sight, smell and taste etc., is synesthetic cooperation in a hinge-dimension.

Figures 1: Drawings from synesthete NH illustrate her number-space perception: an involuntary AND elicited “sensation of presence” of numbers and days of the week in succession.

SYNTHETIC SYNESTHESIA AS PRACTICE

Biograms find aspiring correlations in real time, multimodal digital processing; in the creative experience of intersensory transformation enabled by digital media synthesis. These are commonplace experiences for laptop VJ/DJ artists. They are also experienced in interactive installations that use, for example, parameters of sound (frequency, amplitude, spectral analysis) to control parameters of images and text (size, shape, color, movement, position, rotation, etc.) creating dynamic, image-sound topologies. In media and LiveArts practice, fixed and dynamic correspondences between modal properties are mapped and modulated. The frequent inclusion of random algorithms or aleatory input and/or output methods mediates, within coded parameters, the predisposition of the results. Sensory fusions may emerge between audio, visual, haptic and proprioceptve perception of the event. The perceptual and conceptual smudge. Though a biogram could be considered a specific form or object,^vii it is fundamentally a mode of being of the synesthetic hinge-dimension, ”the form experience takes” (Massumi, 2002, p.186). A dynamic aggregate of perception and memory, a biogram registers the topological shapeshift of continuous change. By way of example, a distributed, networked performance with digital artists hunched over laptops, tweaking images that morph to sonic textures, improvising

Figure 2: Laptop synthesizers in translocal performance 2003, Amsterdam 2003 (Doruff, 2005)

synchronous visualizations of multi-modal transductions, a synesthetic hinge is not difficult to imagine. Metaphorical synesthesia? Augmented physiological synesthesia? The re-emergence of a dormant clinical synesthesia enervated by frequent, repetitive processing? Perhaps the term synesthetic practice best describes the phenomenon. Sensation itself an be re-valued as a vibratory contraction that becomes a quality or qualities – “the concept becomes object as created, as event or creation itself” (Deleuze and Guattari, 1994, p. 211) – feeding back and feeding forward in the experience loop of recursive duration.

Libet’s claim that consciousness backdates itself to produce the illusion of real time synchronicity provokes the argument “that there is no essential difference between perception, cognition and hallucination” (Massumi, 2002, p. 206). If the synesthetic fusion of modalities – synesthetic practice – is hallucinatory, it is as ‘natural’ a function of the affective, modulatory cognitive interval as any other type of thought. Massumi concludes: “The involuntary and elicited no-difference between cognition and hallucination can in turn be summed up in a single word – imagination.” Interestingly, this concurs with Galyev’s assertion that synesthesia in art practice is a “co-imagination,” a comparative mode for non-verbal thought, for “visual and musical thinking.” However and wherever the flightlines of synesthetic research and its variety of interpretation distribute, the interrelations and cross-talk between art, philosophy and science it generates is perhaps its most compelling attribute. Transdisciplinarity itself becomes a hinge dimension, practice as perception.

References:

Cox, Christopher, (2005). Lost in Translation, Artforum, October 2005.

Cytowic, Richard E., (1995). Synesthesia: Phenomenology And Neuropsychology A Review of Current Knowledge,
PSYCHE, Vol. 2, (10) retrieved from http://psyche.cs.monash.edu.au/v2/psyche-2-10-cytowic.html

Deleuze, Gilles and Félix Guattari. (1994). What is Philosophy? trans. Hugh Tomlinson and Graham Burchell, New York: Columbia University Press.

Doruff, Sher ed., 2005. Connected: Live Art, Amsterdam: Waag Society also published online at: http://spresearch.waag.org/papers.html

Galeyev, Bulat, (1999). What is Synesthesia: Myths and Reality, Leonardo Electronic Almanac, Vol.7, no.6.

Harrison, John, 2001. Synaesthesia: The Strangest Thing, Oxford: Oxford University Press.

Holmes, Nicholas P. and Charles Spence, 2004. The body schema and multisensory representation(s) of peripersonal space, Cogn Process, June; 5(2): 94-105.

Libet, Benjamin et al, (1979). Subjective Referral of the Timing for a Conscious Sensory Experience, Brain, 102, 193-224.

Libet, Benjamin et al, (1983). Time of conscious intention to act in relation to onset of cerebral activity (readiness potential): the unconscious initiation of a freely voluntary act, Brain 106, 623–642.

Massumi, Brian, (2002). Parables for the Virtual: Movement, Affect, Sensation, Durham: Duke University Press.

Ramachandran, V.S. & Hubbard, E.M. (2001). Synaesthesia – A Window Into Perception, Thought and Language, Journal of Conscious Studies, Vol. 8, 3-34.

Ward, Jamie and Jason B. Mattingly, (2006). Synesthesia: An Overview of Contemporary Findings and Controversies, Cortex, Vol 42, Issue 2, 129-320.

Whitehead, A.N., (1929, 1969). Process and Reality, NewYork: The Free Press/Macmillan.

i American Synesthesia Association description: http://home.comcast.net/~sean.day/html/definition.html

ii Synesthesia researchers such as Richard E. Ctytowic would narrow those parameters. For Cytowic synesthesia has Cytowic cites five conditions as necessary components of a true, idiopathic synesthesia paraphrased below (2002): 1) Involuntary and elicited: synesthesia is evoked at a preconscious sensory stage; 2) Spatially extended: synesthetes describe going to a certain “place” to examine a sensation; 3) Consistent and generic: modal associations (e.g “A” = red/orange) remain the same throughout life; the “forms” are simple, not complex (e.g. geometric shapes, hot, sweet, etc.); 4) Memorable: the added information of synesthetic experience enhances memory; synaesthetes often exhibit photographic (eidetic) memory; 5) Affect-laden: generally pleasurable; laden with emotional affect

iii Cox writes in hisArtforum article “Lost in Translation”: “Wary of the attempt to reduce sound to sight, Nietzsche insists that the visual and the auditory constitute separate spheres and that the relationship between the two can only ever be a matter of translation or metaphor (in theetymological sense of “carrying over”) that bears the traces of an unassimilated remainder. For Nietzsche, the distinction between the metaphorical and the literal is simply that the latter no longer acknowledges the difference that constitutes it, taking itself to be what it represents. Such literalness is a chief characteristic of the aesthetic discourse of synaesthesia today.” (2005)

iv Whitehead: “A feeling […] retains the impress of what it might have been, but is not. It is for this reason that what an actual entity has avoided as a datum for feeling might yet be an important part of its equipment. The actual cannot be reduced to mere matter of fact in divorce from the potential.” (Process and Reality, 1969, 265)

v A type of transformative superfigure defined by vectors rather than coordinate points. Wikipedia defines a topological space as “mathematical structures that allow the formalization of concepts such as convergence, connectedness and continuity.”

http://en.wikipedia.org/wiki/Topological_space

vi Proprioception is a self-referential interoceptive sensing of the body’s relation to its own movement and to the relative position of its limbs.

vii Synesthetic forms, as new media “objects” might be analogue and tactile (three-dimensional objects modified through digital protocols or digitally screen-based. Screen-based biograms most often comply with certain synesthetic conditions in which intersensory cross-referencing takes place in the non-Euclidean space between the artist and the monitor surface corresponding to Cytowic’s peri-personal “limb-axis space, immediately surrounding the body” (1995).