form+zweck 22:
Tangibility of the Digital - Die Fühlbarkeit des Digitalen


Jörg Petruschat

Keep Cool. Some Remarks on Immersion


Many are still going to great lengths today to see the computer as nothing more than a tool. A prosthesis which serves the purpose of accelerating or, in conjunction with powerful effectors, of amplifying many times what we humans can already do. Often this tool metaphor also proves to be useful, if access to the digital world ought to be configured in a sensorimotor fashion. That which design must provide for this, appears to be already defined in advance by the forms of human physiology and its properties.
But are analogies with human physiology; are hand tool metaphors adequate and suitable for unlocking the complexity of digital possibilities or for perceiving it with the senses? Or does the differentness which can be embodied by digitally generated reality remain essentially blocked to access based on feeling and intuitiveness?
In his text, Jörg Petruschat points to basic connections in human attempts to construct meaning, imagery and multisensory approaches. He sees knowledge about their functioning as indispensable, if we are to use digital technology to go beyond the known regions of mere imitation of the (analogue) world in which we normally live. In contrast to all of those who believe that no eyes or hands are needed any longer, because it's smarter to slide computer processes directly into brain processes, Petruschat sees the embodiment of an ability to investigate in the physiology of the human senses, without which consciousness would not become aware of anything. Even in the exploration of possibilities which lead into the new and the unknown aspects of digital worlds, people are reliant upon these very complex capabilities in the interplay of all of their senses.
Jörg Petruschat has been involved in the conception of form+zweck for many years. Also the subject of this publication hearkens back to his inspiration. He works as a professor in the Faculty of Design in Dresden and is Head of the Institute for Innovation and Design (i_id).

Whenever the subject of virtual reality is raised, the conversation rapidly turns to immersion. What is meant by this is the diving in of users into these realities. The concept of immersion describes an ideal and inside of this ideal there is a desire: the more natural a virtual environment seems to the user, the more familiar it appears to him, the more unquestionably and thus more intuitively would he succeed in acting and making decisions in that environment, one would hope. Therefore, acting in the "real" world is always the gauge in that context. There, in our everyday lives, we would also often act and make decisions intuitively. And thus much is being done to make this immersion so perfect, that there is as little as possible to choose between virtuality and reality.
Many are convinced that metaphors are a powerful tool to facilitate intuitive action in the realities of machines. The problem: Metaphors are thought to be instruments of clarification and recognition. Their utilisation in the context of technology controlled by humans and computers results therefore in an ambivalent situation: On the one hand, they simplify access to digital technology, by allowing it to appear familiar. However, that is not clarification, but it results in a reduction of complexity. On the other hand, however, the use of metaphors in the design of interactions ought to help us to unlock the special performance capability of digital technology. However, the question then arises as to whether a metaphor from the analogue world is at all suited to representing interactions with the digital world and to inspiring the user to explore their specific features.




The word immersion suggests, that when we are in contact with virtual realities (and each machine is nothing other than the embodiment of virtual reality), we are talking about a process which is similar to delving into a foreign culture, somewhat like what you experience, if you go snorkelling off the north coast of Africa: thus, a process where the real situation (you're a freezing white European, who has just closed the door of a CAVE behind him) ought to be forgotten in the face of all the overwhelming colours and shapes of plants and animals, which flow in on you. In short: immersion has been conceptualised as processes, where the point is to make us forget that we don't get wet when we dive in.
One is quick to say: An immersion would be ideal then, if we could still produce the feeling for visual, haptic and olfactory floods of stimuli, when we dive into a digitally produced reality and we could really get wet, if the body thus succeeded in conveying the feeling of wetness. However, then we would have to realise that it would be simpler, more exciting and more cost effective to go to the coast of North Africa. Thus what can be the value of immersion exercises of this type?




In order to answer this question, first of all, we should make it clear to ourselves that the image of diving in is false or just merely an image, a metaphor. The point of each immersion is just not having to or being able to travel to North Africa. Immersion is not "beaming". When we talk about immersion, we don't mean "objective processes", but the creation of subjective conceptions. We just imagine that we are snorkelling. In immersive realities we just want to have goggles on and to breathe air in through a mouthpiece. In better immersions, we also move our bodies in the swell of a digitally generated ocean, but in fact we are not snorkelling off North Africa - even if everything looks like we are and much feels like we are.
For many developers, there is nothing more worthwhile than making digital reality so rich in detail, which it is so similar to the analogue world as to be indistinguishable from it - even if this undertaking will always come to grief fundamentally, due to the more minimal data density of the digital realm. Essentially, their effort is frustrating, because it always has to defend itself against an unending reality which is rich in details and is dependent on the next technological step to be able to achieve storage and processing of even greater amounts of data - without a prospect depending on the information density of the analogue world.
I don't wish to dispute, that the function of the most successful substitute for reality can be a meaningful function for many applications. Nobody wants to do without ultrasound diagnostics or to have resolution with the fewest possible pixels. However, machines, particularly digitally programmed machines, embody very idiosyncratic virtual realities compared to biologically organised entities. Their attraction resides exactly in the fact that they are not just identical to the reality which we discover without question around us. And then the problems of understanding begin here: Can metaphors which are based on intuition and a disposition which individual humans have acquired in machine-free contexts be helpful for orientation in these realities and even more for their control? And: Are individual humans, when they enter these virtual realities, still arbiters of their own or merely users of mechanical options? Can something be created from the interaction of individual humans with machines which goes beyond mechanical possibilities (thus possibilities which are predictable in the programme)? And: What is it that humans can bring to the interaction with machines and which (in the process of interacting with the machine) eventually goes beyond everything that was calculated in the programme and limited by it? We can also simplify all of these questions and compile them: What is the experience which awaits us on the other side of the machine, an experience which can only be produced or evoked with it, but not by it alone?




That this notion works (that we're in Europe and snorkel "nevertheless" in the North African Mediterranean) is an accomplishment of our consciousness. It suppresses the fact that we are in Europe. In spite of this, it allows us to imagine that we are in North African waters. Here consciousness means extracting a meaning other than what is actually present from the data which enter the body. Human consciousness possesses the wonderful characteristic of evoking conceptions based on data which are given directly and which are far removed from any immediacy. Part of this is also that we can imagine for ourselves that the improbable is more probable than fact.
In order for sight to become conscious, data which are absorbed by the eyes and transmitted to the brain are not adequate. Data which originate from two domains are placed with these momentary data which are coming in: On the one hand from different non-visual sensory channels, on the other hand from the arsenals of the memory. Only these data which are foreign to vision enable the brain to connect what is seen with information which the body knows. Only the networking and synchronisation of current data from several sensory channels with data from experience result in what we call conscious experience.
What we call "seeing" is not confined to the eyes. It is a multi-sensory dataset which is the foundation for the inherently consistent sphere of activity which forms our consciousness. And also memory which is woven into this diversity is multi-sensory in nature.
If we see something consciously, then we "know" about this sight, because what has been seen momentarily and preconsciously is integrated with the data from other sensory and experiential channels and thus is accepted as being "true". Seeing what we feel, seeing what we know about, that we are those who recognise something there which we see as "true", as only the "adjustment" which is classified with data from other sensory and experiential domains, evaluates what is visualized for us and thus leaves it to us to have it become probably conceivable. Perception is not a mere registering of something which we believe to exist "out there" somewhere, as it finally seems to come before our eyes, but it is a complex notion, a generative achievement. Consciousness is always a matter of becoming cognisant. However, this also means: We perceive more than we see.


The integration of data from one sensory channel into data from other sensory channels is indispensable for orientation of the body, perception and action. Therefore, sensory illusions normally work so well, because data which are made available for the interpretation of a situation have been isolated and "shielded" from the data of all other sensory channels.
On these are based the compelling effects of many arts: theatre, opera and film present us with the world's stage, without our being able to step on it. Painting reduces the world to something visible, whose immersive effect is based on the fact that the way into the picture is denied to us. Many people close their eyes when listening to music, because feelings and sometimes images, such as those which are aroused by music, can be more easily imagined if we "concentrate" in this way rather than what would be permitted by the narrow concert seat and the neighbours sitting in rows. Escher's pictures are impressive, because they can guide our eyes, like they would never be able to guide our bodies: The twisted stairs which lead back into themselves are thus only so confusing, because it's impossible for us to stand or to walk on them.
This process of integration of visual data (sensory data) in data clusters from other sensory channels and from multi-sensory memory patterns creates the situation for us in which we find ourselves, of course. This integration of data which is currently entering into stored data contexts creates what we call "meaning".




If only the integration of data from various sensory and experiential domains enable an individual to accept something as being "true" and if only the individual datum is interpreted by means of contextualisation and gains significance, then everything that we perceive is of a metaphorical nature. In its literal sense, "metaphor" means "transfer", "metà phérein" to carry or transfer something somewhere else. Thus, if somatically stored experiences (patterns of behaviour which have already been felt through once in attention or avoidance, and somatic markers) are applied to data which are currently entering from various sensory channels, so that a unified, intrinsically consistent scene can be presented (performed) quickly and effectively for the consciousness, then the term metaphor seems to have been used very accurately for this process.
The imagery of the consciousness, the organised integration of experiential data into the the stimulus material which is currently streaming in is the reason that consciousness is an experience for us, a sensory state, and that we "lose" our consciousness, if our "senses fade away" and we faint. We can only experience a situation, feel it and deal with it "in our mind", because the transfer of experience allows this situation to appear to us in a somewhat meaningful perspective.
Nevertheless, when using the term "metaphor" for what we call consciousness, there is a hitch. If we take the previous explanations seriously and strictly, then we could only perceive namely what we have already experienced before. We could only imagine what we remember. The neurophysiologist, Damasio is quite consistent in this regard and believes that the present is only the remembered past and he expands this definition also to include our possible imaginings of the future. Everything which we (can) imagine consciously is based just on experienced reality.
However, not only is the similarity between a current stimulus configuration and reality which has already been experienced immersed in metaphors, but also the difference, the approximate, the "inconceivable" and the "unfathomable" which are present in the sensory material and flood into any metaphorical knowledge. Metaphors, like I have developed them here, are not hard and fast classifications but blurred in their ability to interpret and in this blurriness lies their poetic quality which generates new meaning.




We can also repress imagery and with imagery the poetry and the "experiential quality" of our consciousness, however, by compelling ourselves to apply logic to our train of thought, as they say, if we attempt to reduce the situation which is provided to us in very complex data, to a linear path which puts all simultaneousness and parallelism aside. The logical process is linear in principle. It allows only simple causally linked steps between trigger points.
An early project by Joachim Sauter, the "Zerseher" [the De-viewer], provides a good analogy for this. An eyetracker is installed above a screen which displays the "Mona Lisa" painting. As the eyes of a viewer begin to roam over the picture, this movement is tracked and plotted simultaneously as a kind of a stroke of a paintbrush on the image of the Mona Lisa. What I see with my rambling gaze, I erase on the image. I "de-see" it.
But logical processes are not only good for "annihilating" complex characteristics of an experience, by spontaneously degrading the metaphorical.
In order to recognise the constructive power of the logical, we must free ourselves first of all from the mistaken belief that logic arises alone from well-placed words or graphic symbols which are connected with operators. The logic, of which I'm speaking here, grows out of the inevitable organisation of our extremities. The movements of my arms, fingers, legs, my eyes and my face are always subject to the same muscular or sensorimotor control patterns. If an as yet unborn child involuntarily sticks his thumb into his mouth and begins to suck it (instinctively) then an oscillating circuit of arousal is closed where the certainly involuntary hand movement becomes a somatically identifiable "cause". Once we are born, we attain our self-awareness in response to the movements of our physiology.



Seeing and touching

The thought that we perceive more than we see still has one more dimension, however. A still very widespread conception of our visual perception process is that we break down the objects which we are looking at into basic geometric shapes to make them comprehensible to us. Plato imagined it in that way and through Piet Mondrian and other artists who assumed an avant-garde position, this geometric psychophysics of perception also encroached upon the modern area of aesthetic training. Still today, objects are broken down geometrically into educational precepts a la Fröbel or Montessori, and also in instructions for analytical drawing and it is thus supposed that for this reason we consciously construct something in accordance with what spontaneous seeing instigates "by itself" in a manner of speaking, while following the all-powerful basic geometric rules of the universe. Donald Hofman is to be thanked for doing away with this nonsense. His book "Visual Intelligence" contains a chapter entitled "Spontaneous Morphing", in which he shows that human vision doesn't spontaneously organise objects geometrically but according to contours, more accurately, according to convex and concave edges into visual wholes, into shapes which can also be put together totally from partial shapes. For these cases, Hofman speaks of transversal penetrations. He lists a series of laws according to which our vision organises the length of our field of perception of convex and concave curves and mainly curved areas of that kind. Naturally, this capacity is "innate". But what does this say? We all know that individual humans must first acquire or to put it more concisely, must learn the ability to differentiate in the visual area. What is it that we humans "learn" or "acquire" there? And what does this "spontaneous morphing" have to do with "learning", with intuition and with consciousness?
The primary access to the world of human infants is not visual, but tactile, not in seeing but in grasping and touching. For this reason, the suckling infant is well-named, because he seeks and finds the source of all of his happiness and all of his existence with the sensory cells, many of which have grown in his lips and his little fingers. If we take a somewhat closer look at such a fingertip, then we will first notice its convex shape which is equipped or enhanced with very many sensors. In German we speak of a fingerpad to also mean a fingertip. In this way, emphasis is placed on the fact that we can reduce the contact surface of our fingers to a point - a first, very light touch. Each push or pull brings other senses into play, as the skin "rubs" on every area and because it is elastic. If a finger strokes an edge, then only specific sensors are always activated in a longish stretched pattern. If it strokes a surface or a groove, on the other hand, arousal patterns which are clearly different from it will be created. The conciseness and clarity of a stimulus pattern which arises on a corner or on an edge, are not only increased by the fact that dense adjacent groups of sensors, which are not activated by the edge virtually "hang in the air" and thus generate an nformational value of "zero". The groups of sensors themselves produce significantly more excitation energy on contours, as Johansson investigated. If several fingertips play together, which is usually the case, a spatial arousal pattern is created and is synchronised in the central nervous system to a context of action.
This synchronisation is supported by a muscular movement, which controls fingers and arms virtually "in one fell swoop" and whose arousal and feedback are integrated in the tactile data. But only along the connection of this multiple-unit, parallel, sensory motor touching of convex, planar or concave occurrences with the still extensively undifferentiated visual impressions of light and darkness which are delivered to us humans through our eyes, only the connection of touch with sight, produces those perceptions, into which the human field of vision lets itself be subdivided.
It is no coincidence, but simply physics alone that edges and surfaces always react to light in a similar way: at edges it always breaks in the same dramatic way of light and dark and "continues" on surfaces. Thus, if people arrange the length of the retinal area into contours, if they draw together areas of light of similar shades of colour which are surrounded by convex and concave edges into a figure which they then perceive as a whole, then some kind of visual laws are not working here alone, but it is just the experience of touching and grasping by hand, which is provided to the brain parallel to "seeing" and which forms and produces the arrangement of the visual world. At this point, perhaps I should point out again, that in this very complex process touching is the "tentative", the "exploratory" moment, whereas seeing forms the recording, "tracing" and thus reflexive moment. Here visual perception has an evaluation function with respect to what has been felt or "grasped". In a certain regard, here we recognise the "visual" as a particular level of feedback of a physical movement and we begin to sense, that this feedback can be a definitive stimulus for consciousness-raising and the reason for this is that our consciousness is primarily a "graphic" consciousness .
Nevertheless, Plato and all of those who followed him were right (but in a different way than they thought): The geometric figures which they assumed to be a constituent part of our seeing, are the result of the geometry of the motion of our fingers, arms, shoulders and torsos. What we perceive visually, is not just added together by our brain from individual forms in the manner of a diligent arithmetician, but it exists as an integrated image of the morphology of selected sensors and effectors.




The basis of our consciousness is our intuitiveness in feeling. Consciousness is a symbolic process and what makes neural arousal clusters into symbols, is its similarity to earlier experiences. We "sense" something, because it seems similar to an experience that we have already had, but we still don't have theoretical knowledge of it and still haven't known it for a long time. Therefore, recognition of what's new is an act which is deeply rooted in perception, in "aisthesis".
We can stabilise this consciousness by shifting its emblematic nature to the outside: the gestures and sounds which are our consciousness, our conscious acting and thinking, which accompany it and call it forth are linked to fetishes, reinforced in languages, in objects which we call tools, "objectified" and "frozen", so that our contact can "warm" them up again and set them in motion.
Are immersions (and the virtual realities into which they lead us) something fundamentally different from fetishes, words and tools which stabilise our acting and thinking and broaden their possibilities? Do they contain spaces of possibility which are peculiar to them?
For a start, immersion distinguishes itself from words, tools and fetishes by virtue of the fact that unlike the latter it doesn't extend into just anywhere, that it is therefore just not "instrumental" and discrete, but situational and comprehensive.
Marshall MacLuhan once made a distinction between hot and cold media with regard to the comprehensive character of constructions of reality. He called media which are rich in detail "hot" and he designates as "cold" those where we, as recipients, must participate a lot, in order to warm them up, meaning, to extract significance from them. "Hot" and "cold" are, like temperatures, nothing objective or calibrated, but are only subjectively definable in relationship to one another: In relationship to a film, a radio drama is cold, but in relationship to a telephone conversation, a radio programme seems hot. The "difference in temperature" arises, as we now know, from the variable use of the past which must be invested in receiving, in order to allow it to become a successful occurrence. Hot media bring in their complete meaning effectively in complex references, whereas cold media must first be thawed and warmed up just through our experiences, so that something like an association in meaning is created.
"Cold" media are less sure. They challenge the recipient to introduce himself and his experience into the interpretation of the narrowly held amounts of information. They are more "liberal" and broader because they can be interpreted in more varied ways. "Hot" media bring the recipients into rather passive positions of interpretation and determine their interpretative work almost completely by means of multifaceted specifications. They are definitive, they take us "captive" and have an almost dictatorial character for our interpretation. In the cinema we sit tight in our seats and there is no alternative to what is being shown to us. With our mobile phones to our ears, we can do this and that and it is a big effort for us to recognise whether or not the person to whom we are speaking is also really "up to scratch".
If we only see the media as a means of transmitting news, then the "hot" media will always win, as the goal of all transmission of news is comprehensibility, thus the question of whether recipients manage to rise to the height of transmitters. Of course this is better guaranteed for hot than for cold media. Cold media are less certain. Therefore, McLuhan recommended, as he wrote in his text of 1963, to the Great Powers of the United States of America and the Soviet Union who were at that time pitted against each other as enemies to install "red" (!) telephones parallel to the diplomatic teletype machines, because the tone still "makes the music", even in statesmanlike utterances and thus guides the interpretation closer which is how it also is in "real life".
From a theoretical media-related perspective, a successful exercise in immersion is to be preferred to one that is less successful. But that is only applicable, if the model depicting success is a news theory which is understood in vulgar terms.
It looks quite different, if it is for an interactionist context. We should be wary of describing multisensory interactions merely by means of a model which reduces communication to the relationship between transmitters and receivers. Such a model is scarcely suited to being able to produce "unpredictability" and "creativity" and the interactions and to grasp them and represent them in the first place.
My point is not to defame all immersion as "bad" if it is rich in detail or geared to focusing attention on a narrow corridor of meaning. For all cases, where the point is that people operate machines, that may even be very good. It guarantees the processes that triviality that machines require in order to function flawlessly.
In addition, we can't simply assert that the fewer the data, the greater the freedom or that cold media are consistently the most convenient means of character formation. The fact that a large amount of data determines the interpretation and permits immersion to be deep doesn't have to inevitably result in the disempowerment of individual autonomy.
The determination of an interpretation still makes no statement either about how many completely original thoughts we can have about it. Books are thankfully ample for us to be able to form an accurate image. (They contain data which are very rich in detail) Only this precision enables our thinking to refine and differentiate itself on higher levels of thought formation. The success of this work is dependent on attentiveness and the ability to concentrate and on being able to make the conceptions which the book produces, the subject of evaluation and modelling. And here, books are superior to films or to radio plays, thus to all those media where the time needed for their reception cannot be influenced. As films and radio plays pull us forward on a linear trajectory and don't allow our thoughts to circle and to spiral up on the "thermal" of our high-flying worlds of imagination. It is the time, more accurately the forced tick of the clock which has a limiting effect here on the development of ideas and conceptions, because it imposes a pace on the recipient.
Our consciousness is based on the interplay of different senses. Why don't we give this diversity in the design of interactions of man and machine which are guided by the senses more lea way, so that we can succeed in getting to know the wonderful world which machines can offer us with their wholly unique, mostly very cool form of intelligence? Getting to know this is a course which is going to take a longer time. We must have the time and the usual opportunity to appropriate to ourselves metaphors for the accomplishment of this course. Therefore, it is not particularly clever to want to see and project something quite different in what is "tangible" or "graspable" than in command driven menus, in the keyboard and mouse. In addition, these are only metaphors, after all. Focusing on "tangibility" and "interaction" is an effort, to reach beyond the limitation of metaphors of literalness and two-dimensional hand movements which were inspired by less complex hardware and software capabilities. An interactionist approach will make the designs of the machines, their hardware and software but also old positions of power in the development of new machines available. The concept of interaction does not tolerate concepts that work on the "interface" is a burdensome exercise on the outside of a machine whose core was conceived by a few clever humans, but which, unfortunately, must be sold to many stupid people.
The multi-sensory metaphors of our consciousness are geared towards exploration and arose in analysing the complexity of a reality into which we have grown. We should have confidence in the performance capability of this complex, multi-sensory system. Here is a broad and interesting field of research, experimentation and investigation, to help us to understand what it is that is irreplaceable in our lifestyle which can be provided by interaction with digital technology.