Tangibility of the Digital - Die Fühlbarkeit des Digitalen
Hermann Wolfram Klöckner
Object and Medium
If we want to develop depictions of three-dimensional ideas, then we use screens or drawings. If it's a question of clarifying relative positions and proportions, then models and sculptures are made. Rapid prototyping processes have been helpful to us for a long time in making data which have been created in virtual environments without any manual process, spatially tangible, nevertheless.
However, what if our ideas are dynamic? What if we don't want to convert them first into two dimensions in the form of animations or films and thus degrade them, but want to put their „information" directly into a three-dimensional object which will represent them sequentially, at least? Must we climb into a CAVE for that purpose and connect our receptors so closely to an electronic system that our everyday situatedness disappears?
Hermann Wolfram Klöckner developed an object and a process, TransForm, as a part of his exhibition concept, Xplain, which enables us to receive information in three-dimensional and sequential forms, without having to climb into a CAVE. Klöckner thinks that with this, he has not only created an object for interactions which gracefully avoids the problems of immersion, but a new medium, a new form of communication.
Hermann Wolfram Klöckner is a product designer and has been working since 2007 at the ID 5, Institute for Product and Process Design and at the Institute for Time-based Media. Both institutes form a part of the Universität der Künste (University of the Arts) in Berlin.
In my work as an interface designer, spatial anchoring of information forms one of the central themes.
The means utilised are not used as "windows" into another reality, but serve to expand the communication of the physical environment on site.
For this purpose, objects and locations are joined together with media resources in to form an integrally designed situation where dialogue is involved.
Communicative expansion of the familiar reality definitely helps to facilitate deep immersion in the medial experience. As the usual physical world is not abandoned at first but merely expanded, it supports the credibility of the situation with all of its familiar idiosyncrasies.
The struggle for the most intense "immersion" possible which is characteristic of cyberspace - as deep a delving as possible into a medial experience - is charmingly circumnavigated in this way.
In order to discuss such a concept as communicative enrichment in concrete terms, the exhibition concept "Xplain" and "Transform", the completed exhibit which it contains are presented below.
Xplain is a concept for an exhibition about the world of topology and form which surrounds us and of which we ourselves ultimately consist. In Xplain it's not a question of an abstract mathematical perception which is unique to experts, but its association with the everyday world of those visiting the exhibition. How do these forms emerge and what features do they have? To what forms do basic geometric operations such as rotation, scaling or reflecting lead? What geometry is inherent in existing forms? What emerges, if we make free forms?
In the same way as in a conventional exhibition, it's a question of collecting and understanding exhibits. In contrast to traditional presentations, in Xplain, the visitor can gain access to the information interactively through the exhibits and methods are provided for him to enable him to acquire further knowledge. In a further development of the concept, it would be conceivable to connect the exhibition with a library in such a way that a trail projected on the floor would lead from the exhibit to the reference book as a direct link. Or, like in Semapedia, where the exhibits are provided with a barcode which contains their individual Web addresses. If you take a photo of the barcode with a mobile phone, you can reach the internet page of the object and can learn, for instance, that the icosahedron is one of five Platonic bodies and what these have to do with Kepler's cosmic model.
In order to achieve an effortless connection of the various levels of knowledge in Xplain, within a linked overall installation there are nine stations which are arranged as an intensive dialogue between the exhibit and the visitor. In contrast to customary exhibits in science centres where a defined action is triggered by pressing a button or a specific task ought to be triggered in accordance with instructions given, here a wider scope of action is defined and the visitor is accompanied continuously in his exploration of it. A communicative process emerges and facilitates the acquisition of knowledge on an intuitive level which is enriched with meta- information.
Xplain's interactive concepts connect thus to familiar patterns of action and expand these medially. The interface responds communicatively to visitors and doesn't expect any "operation" from them.
Thus, some of the exhibition's content is accessed by simply moving in the free surfaces between the exhibits:
A reactive room installation on the connections between two-dimensional and three-dimensional forms presents basic geometric structures as an interactive floor projection. These projected objects behave in a seemingly curious or playful way, by approaching the visitor, accompanying him or crossing his path. The visitor can capture the structures and obtains the geometric development of the object (the two-dimensional plan of the structure) by stepping on it undauntedly, as well as text commentaries about the targeted object on site. Information sought in the process ("Pull") is received considerably more intensively by visitors and especially more joyfully than something which is directly offered ("Push").
Each experiment is a question to nature - An acquisition of knowledge through dialogue
In order to convey facts in a playful manner, in my opinion, more than the design of an elaborate interface is needed as a replacement for a simple switch to activate pre-fabricated subject matter.
The experience in an intensive situation involving dialogue between the visitor and the exhibit is richer and more multifaceted. The role which information technology can play in such a dialogue is a supporting albeit decisive one: It can relay, emphasise, explain, instruct and comment. So a deep communication situation is created which is moderated rather than guided by the design of the exhibit.
Following this creed, the free-form exhibit of the Xplain presentation was developed: Here the visitor can conduct completely free shaping experiments using quantities of conventional modelling clay. During this process, the forms which emerge are analysed by a special camera system and are provided with a rear projection in real time. In this rear projection, characteristic properties of form such as surface, volume, topological structure or curve progression are reproduced. This mathematically precise analysis of form is connected directly with a completely free playful shaping process.
Thus, an installation which is characterised by detailed feedback facilitates the connection of a familiar situation with a totally unfamiliar expansion.
However, the richest communicative situation arises, if the exhibit inspires the visitors to have an intensive dialogue with one another. The concept of the exhibition makes it possible for all exhibits to be used simultaneously by several visitors.
Using the exhibit station for the cutting of several shapes, the "Boolean exhibit", we can illustrate well such a dialogue between the visitors themselves and between the visitors and the exhibit. Using this Boolean exhibit, the visitors can learn, how a third structure with completely new properties is created by penetrating two structures. For this purpose, two laser scanners project basic geometric structures such as cones, cylinders or pyramids into a liquid. Where the differently coloured projections penetrate, a visualisation of the section structure is created by the addition of a colour mixture. The shapes and properties of this structure are therefore largely dependent on the structures being cut and their angles to one another.
Visitors now can control the process together in all of its parameters and receive context-sensitive commentary about the emerging structures.
The exhibits presented use the familiar physical world to achieve a high degree of immersion and credibility in communication.
The natural intensive immersion in what is happening is successful, because the medial enhancements are only superficial and a commentary. The material properties of the exhibits remain unaffected throughout. The completed TransForm exhibit undertakes an essentially more fundamental intervention. Here, a structure which has been divided into partial triangles and has been affected not just superficially but in all of its materiality is used as a means of communication for the presentation and manipulation of shapes. TransForm facilitates the physical exploration of metamorphoses between three-dimensional structures. In this way, the object itself becomes the medium. The exhibit interacts with the visitor by using its whole form to communicate with him.
When there is such a fundamental intervention of media in the physical world, questions about the effects on the process of communication between the visitor and the exhibit are raised. Is such a distorted artefact still credible and is it possible to have an intuitive interaction with it? How deep can the immersion in a media-related experience which moves far beyond the frame of reference of the visitor's experience be?
In order to discuss this question, using TransForm, a concept of immersion which is applied to tactile interfaces ought to be examined below.
TransForm" is in its present phase of development is a spherical, flexible object, approximately 30 cm large, which is capable of assuming various forms and can itself be freely distorted by the user. In contrast to known tactile media, such as data-gloves, which latch onto the users' senses, a physical artefact which is more communicative due to its modifiable form should come into existence here.
If we examine established, graphic output media, then the following image of the demarcation between medium and content is the result:
If we want to represent two-dimensional content in structural terms, then we use a printer.
Two-dimensional dynamic content can be shown in displays and with projections.
Producing three-dimensional structural content has become virtually commonplace due to advances in rapid prototyping.
However, if we want to represent dynamic content in three-dimensional terms, then we are reliant on a more or less successful sensory illusion on the part of its users: Starting with stereoscopic displays and proceeding via holographic processes all the way to data-gloves, the ambitious attempt which is common to them is to fool our senses in as detailed a way as possible through a mostly direct latching on of the medium to our receptors and to what excites them, in order to facilitate a certain degree of immersion.
In contrast to that, the attempt should be made with TransForm, through the use of an extremely low-resolution, object-like medium to achieve a very high degree of immersion. To this end, the receptor-based approach was replaced by an object-like one. The "trick" is never to leave the physical world in the first place. Immersion is a yardstick which we do not apply first to the natural world, as it forms the reference of experience, by means of which we measure our media. Now, if we use the properties of physical objects, such as weight, surface structure and temperature which are present anyway, then the question about immersion is not asked any more.
What suggests itself even more strongly is the question as to whether, with such a dynamic artefact, we are still dealing with a medium at all. In order to discuss this, the structure of TransForm is described in more detail below.
Object-like tactile interfaces
Even if the creation of a tangible medium is established as a requirement in advance, this does not necessarily have to mean the creation of a volumetric structure: When analysing the perception of objects around us it is noticeable that they generally are presented only as shaped surfaces - the volume under these surfaces is added in anticipation by our experience: If we slice a loaf of bread, what was previously hidden volume is revealed now as newly emerging surfaces along the cut. Merely semi-transparent objects enable us to have a more direct perception of the volume due to refraction and diffusion of light inside of them and on their surfaces. The volume itself is nevertheless only to be experienced indirectly. In order to construct an object-like medium, it is sufficient to create curved surfaces.
Virtual, triangulated surface models are known from geo information systems, for instance, in computer graphics and FEM calculation in engineering sciences. The decisive step in the designing of TransForm now is to take this method from the virtual world and bring it back into the physical world. The models for that were known structures like the folding of the skin of a tick, the malleable perforated metal plate and sub-areas of origami.
However, the object-like quality is achieved initially through the topological closing of the modelled area for the modelled structure. In this way, a triangulated approximation of a sphere is offered in the form of a tetrahedron, an octahedron or an icosahedron. However, in order to achieve adequate resolution, TransForm is based on a further subdivision of the icosahedron into a geodesic dome as described by R. Buckminster Fuller. 2
The number of these subdivisions determines the resolution. It decides, among other things, how we evaluate the quality of the interaction with a medium. The pixels in the raster image on the screen or in a camera have their equivalent here in the individual triangular sections. In receptor-orientated tactile interfaces (somewhat like data-gloves), the degree of immersion is significantly dependent on the resolution which has been achieved in the tactile output and input. Above all, if we consider that every human fingertip contains approximately 700 pressure and touch receptors, on average, each receptor would have to face an actuator which is attractive to it for the convincing reformation of natural surfaces.
Like our relatively low resolution eyes due to constant "grazing" in our field of vision and the constant arrangement and combination of momentary impressions gained or "blinks" in order to put together a high-resolution temporary overall picture, touch is also significantly characterised by a dialogical searching process. This is even reflected linguistically in terms such as "to feel out", "to feel one's way" and "to feel". Exactly these stroking touching movements are to be replicated only with great difficulty by receptor-orientated interfaces which latch on directly to the senses, as the interface which is latched on cannot move in relationship to the receptors. In addition, material properties which are perceived in various ways are based definitively on the different ways these materials perform when they are subjected to the friction of stroking touch movements: because of their microstructure, materials like glass, plastic or wood each behave very differently when subjected to static and kinetic friction compared to the human skin surface which is feeling them. Object-like tactile interfaces like TransForm are admittedly not capable of exploiting these properties medially, but they have their own surface characteristics due to the materials used in the structural form.
It could be argued, based on this, that object-like tangible user interfaces provide only a low, spatial resolution, as there was a higher degree of immersion in any event, due to its materiality. However, low resolution naturally limits the range of content which can be represented.
Now however, tactile interfaces regularly use a considerable amount of precision engineering, which grows with increasing resolution. When developing TransForm, I made the presentability and recognisability of the human face the basis upon which the medial quality was evaluated. The portrait of Abraham Lincoln which was published in 1973 by Leon Harmon (Bell Labs) in Scientific American served as a visual reference. In this depiction, the likeness of the erstwhile American President is only clearly recognisable in a resolution of 18x14 pixels.
One would think that the representation of human faces would already be making relatively high demands on the capabilities of a medium, particularly a tactile medium.
Therefore, TransForm was converted into a third frequency geodesic dome, thus with a resolution of 270 separately controllable sections or 180 individual triangles. This function results from the combined working of approximately 8,200 individual precision engineering and electronic components and a simulation and control software programme which was developed for this purpose. Essentially, the star-shaped arranged components made out of so-called shape memory alloy (SMA) which can be varied in their dimensions by means of the appropriate modulation of control signals and which find their mechanical "opposing muscle" in TransForm's latex surface. The use of 270 bending sensors with the same resolution is anticipated to have an impact the object. The spectrum of forms resulting from the geometry used will range from basic geometric structures like tetrahedrons or octahedrons up to a guessable, although not identifiable face.
However, the credibility of a medium is essentially drawn from the congruence of different sensations. Therefore TransForm modulates the visual aspect of the appearance beyond the pure shape. For this purpose, volumetric material textures are calculated and the information corresponding to the current form of the object is displayed on the outer skin. Through the use of simulation and control environments, virtual light sources with appropriate illumination of the triangular section of the surface can also be put into effect. All of this serves to create a coherent link between the tactical and the visual communicative experience.
The quality of the interaction achieved by means of TransForm is already characterised by a great amount of credibility at the attained level of development. However, the low spatial resolution limits the variety of the presentation.
In addition, the temporal resolution of the medium is still restricted and quick changes of shape such as lip movements have not been able to be represented in real time up to now as morphing requires approximately ten seconds, depending on the particular shape. However, here as well, due to the object-like quality of the medium there is coherent, credible communication with the object, as the slow movements of TransForm become smooth and fluid due to the natural material properties, similarly to the setting of keyframes in an animation.
The interactive concepts under discussion here in a museum context are not necessarily linked to an exhibition space. In my opinion, the obvious location of information by means of communicative expansion of the physical world facilitates the design of more inconspicuous communicative situations, in which a high amount of immersion and intuitiveness emerges as a matter of course. Thus an unpretentious execution in contexts removed from museums can display its practical relevance for example in direct, location-based, interactive instructions.
If we take the concept of tangible media like TransForm further, then through the use of other materials in combination with a dramatically heightened resolution a kind of digital semi-finished product can be created: a product could thus be produced locally through the charging of a licensed design to a semi-finished digital product. As a final consequence, even the recycling of things will give way to your on-site updating metamorphosis by metamorphosis.
Nevertheless, the question being asked right now with regard to tactile interfaces like TransForm is whether we would not greatly increase the "information load" of our environment through this complex media-related playing with a further sensory channel. With this in mind, let me consider the "information load" during a walk in the forest: The multifaceted visual impressions, the fragrance of the conifers, the twittering of the birds, the sound of the leaves, the soft yielding forest floor under my feet - I don't take all of that as an information load, but rather as an information joy, truly.
So, if we strive to implement tangible media, in my opinion, this will come about essentially in a well-crafted and carefully choreographed design, not in order to replace the physical world but to enhance it in a charming way.
Contact: HYPERLINK "mailto:firstname.lastname@example.org"email@example.com
1 FEM (Finite Element Method) a method for the calculation of material and structural properties, by which the geometric structures to be examined are split up into many finite sub-cells (finite elements).
2 Your Private Sky - The Art of Design Science" p.354 ff, Published by: J. Krause, C. Lichtenstein, Verlag Lars Müller, Zurich 2001
3 Leon Harmon, Scientific American 11-1973, pp. 71-82, "The Recognition of Faces",