Tangibility of the Digital - Die Fühlbarkeit des Digitalen
Christian Zöllner has devised an arrangement and designed tools for three-dimensional sketching. This is unusual and new but not possible without a virtual environment. His intention was to try out and implement a rather broadly defined concept of "Tangibility" through which not merely analogous, i.e. psycho-physiological objects and their processing are translated into digital, i.e. "units of information" (or vice versa). His aim was to open up, in the sphere of designing, creativity that is centered in spatially concrete actions of the body and to achieve a design quality in its actions (and to make it accessible to further processing) which is not accessible to conventional 2D sketching on plain paper by hand and in mostly a rigid sedentary body posture.
The arrangement that I designed for three-dimensional sketching did not emerge without some presumptions. Therefore, following a few preliminary remarks, I wish to first discuss three forerunners of development which are similar to my project and have opened up the decisive scope for ideas which I intend following up.
Different modes of interaction with the computer at present reduce the human being mostly to eyes and index fingers. The body with all its abilities and senses is hardly challenged. For the designing process, this means a limitation of the power of aesthetic judgement and creativity. In this respect, Tangible Interfaces can accomplish amazing things by bringing back the whole-body component into the designing process and by creating divergent motor activity as well as distinctly purpose-orientated forms of interaction.
However, the situation today is something like this: the operator sits in a mostly uncomfortable position in front of the desk, the lower arms resting on the tabletop with one hand over the mouse and the other on the keyboard while the view keeps shifting between the upright image on the monitor and the array of alphanumeric symbols on the keyboard. We assume this position when drafting a text, editing digital photos or designing machine parts. Although the purpose of these actions may be different, the cognitive and motor activity remains the same. But we know from our experience with artistic designing how heavily the generated form depends on the actual execution that accomplishes it. Therefore, there is no doubt about the fact that a standardisation of motor activity and cognition - which is thrust on us by the keyboard and screen - absolutely degenerates the possibilities of designing process. A computer not only "computes" various applications and services within, but also determines the manner in which its processes are written and manipulated; that is to say, it also limits the capabilities of the user in front of it. The direct interaction with the whole body, the spontaneity and the innumerable abilities we possess through our hands, arms and our body posture remain underutilised and reduced to mere pushing of the index finger back and forth.
But first things first
Computers that are used nowadays in the field of designing are not exclusively made for designers, but rather for a general and broadly defined use. As much as possible, we should be able to perform with these devices the processes of recording and verifying data in a most universal and simple way. Meanwhile, classic input devices such as the keyboard and mouse are joined by piano keyboards, guitars and also turntables (as input devices in the field of music). Sketchpads and pens enable through their graphical creative processes transformation of emotional expression into information - something that is in principle not possible with a keyboard. Particularly artistic forms of expression are based on a versatile, special and well-trained motion of motor organs. As computers succeeded in setting up a sketching surface and pen to translate the analogue reality of drawing into digital data, the designer was no longer forced to adjust himself to other strange interfaces, but was able to express his ideas in an accustomed and established mode of work and not in a forced and degenerated form of work. The archetypical operations and the consequentially applied artistic means, particularly the tools of creative processes like sketching with a pen or plucking the string of a musical instrument, will not be discarded or replaced by these interfaces that imitate the analogue environment, but are integrated and augmented by these digital innovations. Thanks to MIDI, Firewire or USB interfaces, which simplify the use and alteration of different activity-related interfacing with the personal computer, today a new quality in designing has been achieved. Designers, musicians, artists - who once used an interface that was capable of capturing intuitive and aesthetic components of action for preparing these for digital processing - can hardly imagine going back to the old keyboard standard. These intuitive and aesthetic processes are particularly interesting and valuable in various phases of ideation and concretisation of ideas. The point is to bring means of solutions, which were unknown till now, through recording motor processes operating spontaneously or without parameters into the light of awareness and into the reality of communication.1 The development of concepts of innovative interface and interaction can be helpful in widening the scope of the personal computer during the phases of inspiration, ideation or sketching and designing in order to make the creative potential accessible and expressible more easily. This will invariably lead to a consistent integration of hardware and software.
On the basis of the following three examples I wish to show which ideas have been there - even before my project - to support the designing process commencing with generation of ideas and moving on to elaboration of concepts and evaluation of initial ideas and form or shape perceptions through Virtual Reality and Ambient Intelligence.
The "Cabinet", developed at the Technical University (TU) of Delft, is a digital enhancement of the traditional mood board. Mood boards are used by designers to visually illustrate through picture collages certain facts to themselves, co-workers or clients. For this purpose, pictures, notes end even whole objects are fastened to walls with magnets or pins. They can be arranged in groups and annotated using pens, coloured tape or Post-it notes to create an interrelationship between the images. This means, mood boards serve as inspiration and map an area of problems with visual references and help delimit and resolve them. Adriaan Ianus Keller and a team of designers and scientists from the ID Studiolab of the TU Delft intended to make the process of creating such mood boards or "portfolio walls" faster and more intuitive, especially by considering the wide range of possibilities of digital data processing. In this, working with the PC is significantly less than working directly with and on the images. Keller's digitally supported mood boards are collages fastened to the walls (he calls them "context") which were first edited on the desk (he calls it "layout") and finally provided with notes or small sketches (he calls this "concretisation" phase "precision"). Context, Layout and Concretisation are, therefore, the three levels of interaction or the "three ranges of interaction".
The "Cabinet" integrates and combines these ranges by a simple technology: pictures taken from magazines or books as well as sketches and objects can be placed on a tabletop, into which a Wacom display is incorporated. A camera mounted over this setting captures the arrangement. If the objects are taken away from the surface, their digital image continues to be projected on the tabletop through the Wacom display. Now, these "digital shadows" can be selected, edited and rearranged using a pen-like device. The pen is used not for drawing or writing; it functions more as an extension or direct translator of movements of the user's index finger. Similarly, images previously stored in an image database on the hard disk can be loaded into the "Cabinet". Unlike the already established mood boards the "Cabinet" is aligned not vertically but horizontally. This enables spatial objects to be photographed in a better way and paper sketching to be manipulated directly on the mood board. The photographed objects and images can be categorised into topical groups or stacks and fanned out again by short tapping-like commands with the pen. Through the digital projection the images can be animated and are enabled to move freely on the tabletop. If the images come close to each other, new and unexpected contexts can occur among them. Precisely this aspect makes the Cabinet very resourceful in the early phases of designing process, because particularly in this often divergent phase new contexts need to be anticipated and explored.
Three-dimensional in two dimensions
When the first phase of identification and visualisation of the problem area is completed, the next phase of generating ideas through sketches, smaller and less complex paper or cardboard mock-ups starts. Product designers, industrial designers and architects use often sketches to quickly view their ideas, do trials or corrections with the pen. Or one may create sketches as illustrations for others, often to support a brainstorming session during which problems are identified, discussed and graphically solved. Sketches help us to generate ideas, because they can be corrected and evaluated over and again simply by overlaying methods. An idea that is not yet finalised can evolve graphically into a sophisticated brainchild that will ultimately turn into a perfect design during the subsequent employment of CAD. Though this drawing-like sketching has many advantages, it is bogged down nevertheless by some constraints: objects which are spatial always appear in perspective truncation on the plane surface of the sketching. Their three-dimensionality is just an illusion and the third-dimension is merely converted into the second. The third dimension as an area of action is not accessible to the graphical sketching.
As a result, mock-ups and model making become relevant. But: the additive and subtractive processes of mock-up and model making lack the very facileness, fleetingness and ambiguity that are integral to a normal sketch. Since the mathematical-geometrical reconstructions of the analogue world of shapes can be manipulated on the screen directly and through gestures without the command language dependent programming, the idea is to provide formal data to the designing process also in three dimensions. At this point I will skip all the trials that visualise or represent the already completed construction data in 3D or true to scale environments. I am concerned about the tools for producing and modifying (not just displaying) shapes in three dimensions.
Virtual Tape Drawing
That being so, a team belonging to the software firm "Alias Wavefront" headed by Bill Buxton and Tovi Grossmann of Toronto, Canada, developed a "Virtual Tape Drawing" which they presented to automobile designing companies worldwide for testing in 2001. [Figure ...] Their approach was to develop an alternative interface for 3D-modelling on large displays. The team designed, based on the analogy of tape drawing, two interaction tools that mutually acted like one hand holding a roll of adhesive tape and dispensing the strip and the other fixing it on the wall. The person operates ("tapes") in front of a large screen, on which a beamer projects a perspective view of a previously made CAD model of an object in 1280x1024 pixel format. Although the three-dimensionality appeared "only" as a plane projection, the designer was enabled nevertheless to work on a three-dimensional model and have control glances on the projection on the wall in the space in front of it. For this purpose, the three-dimensional view was split into two-dimensional construction layers which the designer could select to draw strokes "over them." In fact, the designer operates in free space without the frictional resistance from the wall. To capture the designer's hand movements each interaction tool that he held in his hands contained a Flock of Birds tracker with a simple switch which allowed him to switch the tracker ON or OFF and thus to generate or interrupt the virtual "adhesive tape". The position of the tracker was displayed as a cursor. Although the use of the cursor on 2D planes was strongly reminiscent of the use of mouse, the advantage of the project was in the fact that not just the index finger and wrist but the ability to use the whole body, especially both the arms, for creating the lines.
Another more recent approach comes from Pedro Santos and André Stork of the Fraunhofer Institute in Darmstadt, Germany in collaboration with Rodrigo Filipe and Joaqhim Jorge of the Technical University, Lisbon. The aim of the research team was to accelerate the designing process and close the gap between manual designing and digital finishing. Instead of photographing first the "taped" design, i.e. undertaking manual corrections on large-format prints, so as to translate them later into digital data in CAD programs through Wacom Tablets, the manual corrections should take place already in CAD. In addition to the plane taping of strips on section layers, this version of "Virtual Tape Drawing" should enable working in three dimensions. Such a procedure could be well-suited to show modifications on clay models or imported model parameters.
For tracking the movements optically, the team from Darmstadt and Lisbon used rings which the designer simply wore on his index fingers. [Figure...]
Now, if a virtual 3D model created through CAD requires to be "redesigned", one only needs to move the ringed (tracked) index fingers along the object lines to improve them.
A virtual line spanned across the fingers, which was created by the leading hand (mostly the right) and its other end reached the following hand (mostly the left). The team calls this method "gadget free", because there is nothing to be held by the hand. In this operation, the interface is controlled, instead of actions, by gestures that are similar to those in analogue tape drawing. The interface is less "graspable" since the gestures executed by the designer only indicate a movement. They relate to the actual action of the "tape drawing" as a symbol does to the character it signifies. Whereas in the early version of the "Virtual Tape Drawing" by Alias Wavefront the designer at least rudimentarily "emulated" and felt the real tape drawing with his interaction tools, in the current version the designer only indicates the "taping". By dispensing with the interaction tools which can be held and manipulated by hands, not only a different feeling is accomplished, but also another, stronger reflective correlation to the (creative) process.
A virtual world of tools for designing
Johann Habakuk Israel, a computer scientist at the Fraunhofer Institute for virtual product development in Berlin, has developed a technology that enables spatial sketching in virtual environments. In this case, the virtual environment is a CAVE (Cave Automatic Virtual Environment), which is something like a Holodeck that most of us know from the science fiction series Star Trek. In a CAVE, a spatial image is generated for the relevant position of the viewer by using a 5x stereoscopic projection (behind each translucent room and ceiling wall is a projector) which he sees in front of him through specially made goggles. This technology has been already in use in the fields of machine and vehicle designing and Usability Tests. However, it is only used for reviewing CAD models previously created on the PC. Israel developed for CAVE a Sketch Application. It supports a transducer, which, like a pen, generates a stroke of line, when moved, and stays suspended in space like a weightless thread. One can go around this spatial stroke, observe from all sides and extend it into a more complex sketch. Whereas a conventional sketch is merely a graphic projection of an imaginary object or conceptual model, a "spatial" sketch embodies all projections of the object, because the viewer can view the sketched object from any perspective.
In the CAVE, the tools as well as the designer's position are located by means of magnetic tracking where the advantage is that no cumbersome antennas need to be fixed to the tools as against optical tracking. But the drawback is that the accuracy of location is low and the cable leading from the tools to the computer can be obstructive.
I have tried to make this new technological option useful to designers as intuitive and uncomplicated as possible and to widen its creative potential for designing. Consequently, this led to the questions: How should the interaction tools look with which spatial sketching is done? What are the requirements of a designer for a drawing medium? And: How can I break through the limitations of two-dimensionality, which nevertheless fires the imaginative powers just through reduction, into the three-dimensional space so that the proliferation of information does not hamper but rather promotes and stimulates the flow of ideas?
The answer is a set of four tools and the related concepts of interaction. As against CAD, the tools as well as the concepts of interaction denote something totally different and self-sufficient and enable designing actions and processes unknown till now. From them one can expect that they will also yield innovative designing results. While CAD is meant for revamping previously sketched, precise and selected draft designs, my suggestion caters exclusively to the creative phase of gathering ideas for three-dimensional sketching. Its strength is in its accuracy and precision as compared to CAD, and particularly its potential is in the far-reaching freedom as opposed to limiting parameters and in the diversity of renditions it unravels.
In all, four hybrid tools have been designed using which a large number of operations can be performed which are similar to sketching in two dimensions. I consider these tools hybrid because, on the one hand, their shape is materially real and indeed physical; but the forms, the strokes and the surface or volume they create are generated virtually. The tools should be as "tangible" as possible, that is to say, "self-evident" in their sensual presence which is why buttons, menus or selection interfaces were dispensed with. This resulted in prototypes with which already test series have been implemented and refined into design samples.
The basic tool for sketching is a pen. In order to draw on paper with a pen, the pen is placed along with the drawing tip on a medium to produce a visible trace resulting from abrasive action. On the one hand, the medium is everywhere in the space and, at the same time, nowhere since there is no resistance. The pen must know specifically when it should produce a track and when it should not. Since buttons or input devices are avoided, which are distracting for the sketching procedure, the "pen" has the shape of a closed quill in which the upper part has a flexible shape. This connection between the material's firmness and flexibility has often the multifarious advantage for handling and controlling the processes in absorbing the forces dynamically. This made it possible to dispense with complex force feedback motors for many applications but still produce the feeling of increasing pressure and at the same time tap through simple force sensors.
If the upper flexible part is pressed against the lower, in the same way as pressing a pen on a sheet of paper, a line is generated corresponding to the hand and arm movement. The force sensor, which measures the pressure under the flexible part, allows not only recognising the statuses ON or OFF but also any intermediate stages. Thus, the pen can be "gently lifted from the medium" so that the produced line becomes thinner and opens up more possibilities of expression and more personality to sketching due to this pressure sensitivity. In contrast to the "Cabinet" and the viewpoint of ID Studiolab I have used the pen exclusively for drawing. I did not want to confuse the power of association that is inherent to the handling of the pen, i.e. to simply draw lines, with the fact that the pen can also be used for interaction with objects.
For the interaction I have designed a second tool by which lines drawn in space can be reciprocally repositioned, scaled or rotated. The Snapper functions just like pliers or tweezers: if a line needs to be shifted, it is simply "picked up" by moving the Snapper towards the line and pressing together its open ends. Now the line is held "in the hand", positioned at the desired location and released from the tool by opening the Snapper. These motions are known to us from everyday life and even from the hands' natural movements and, therefore, can be mastered quickly and need no further explanation. The Snapper too is based on the principle of spring action, only the shape is more open. Since the concept avoided erasing or return functions (sketches basically allow overlays), the unwanted spatial curves can be easily removed from the CAVE with the Snapper or thrown into a virtual waste paper basket.
For all the tools by which surfaces can be created (or removed), I have developed the Shaper. The Shaper is completely a new type of tool, because it combines the concrete property of gripping with the hand and the abstract, namely a virtual tool for generating or subtracting surfaces and volumes coupled with gripping. But what is a virtual tool? Just like the phenomenon known from animation films in which its advocates can draw cudgels or escape routes into their world in any way they like, designers too can create their own virtual world of tools with this Shaper. This operates like this: a shape, for example a circle, is drawn in the virtual space by using the pen at the end of the Shaper. This circle drawn in the virtual sphere is tightly coupled to the Shaper's "real" grip in my hand and with the tool I can move around freely in the virtual space. A circle drawn through the space can be thus extruded into a tube profile (a cylinder mantle) simply by "drawing" the Shaper through the space with the hand. If I draw a broad line at the grip of the Shaper I can use it like a brush or a broad paintbrush and create or fill surfaces, a square can be shaped into a table leg and so on.
In this way, the designer is no longer bound by real and predefined tools which only dictate creation of shapes, but he is enabled to fabricate virtual tools nearly in any manner which allow him to generate specialised shapes. Thus, he can set any parameters for the design even in the early stages of designing easily by picking up new and original tools especially suited to a specific purpose and not from any existing library or toolbox. The Shaper too follows the principle of spring action in its basic concept like the pen and the Snapper. In the case of a Shaper, also the upper flexible part is pressed against the lower to activate a force sensor that triggers a shaping operation, for example, extrusion.
The Bezier tool
Everyone is familiar with Bezier curves from CAD programs and one cannot do without them. Using these Bezier curves, paths of lines can be connected and manipulated through points located outside the line. If these curves are extruded, complex open spaces can be created. It was interesting to integrate into the concept and manually operate this form of display which designers knew it all along. Thus, the outcome is that there are two identical tools - one for each hand - across which a cubic Bezier curve can be spanned. If the hands and their position are changed in reciprocation the curve changes its appearance. In horizontal tool alignment it can assume the form of a nearly straight line or, spatially, that of a loop if the arms cross over. If one of the tools is squeezed, the curve jumps from this tool to the other. And when the tool is relaxed, the curve remains suspended at its position in the space. If both the tools are squeezed simultaneously, the curve turns into a surface in which the shape can be dynamically controlled by moving the hand. In this way, complex open spaces can be created by hands that are otherwise hardly possible in CAD.
In addition to these tools, a number of integrated scenarios have been scripted that help promote, for example, a creative ambience in CAVE environments and initiate creative processes. These include arrangements of image and shape databases and technologies, such as sketches that can be stored or exported individually or embedded in formatted environments, as well as possible ways of trend-setting collaborations and data exchange through VR CAVEs.