form+zweck 22:
Tangibility of the Digital - Die Fühlbarkeit des Digitalen


Eva Hornecker

Tangible Interaction - An Inclusive Perspective


Eva Hornecker makes the case in her article for including all of the senses when digital systems are being integrated into the everyday world: multisensory interactions should help to acquire a new richness in habitual experience and several patterns for interaction. To her it is more important to achieve clarity by means of interactions between humans and machines and, to a growing extent, between humans with the help of machines, than to (merely) design objects and to optimise "tiles" or "props" which convey these interactions to devices.
Eva Hornecker presented a framework on the subject of "Tangible Interaction" to the CHI (Conference on Human Factors in Computing Systems) 2006 which received much attention and participated in the organisation of the first TEI (Tangible and Embedded Interaction) for 2007 in Baton Rouge. After spending some time on research in Austria, New Zealand and Great Britain, she now works as a visiting research fellow in the Pervasive Interaction Lab at the Open University in Milton Keynes.

Computers are increasingly leaving the desktop - they now are embedded in our environment and into everyday objects, moving out into the world. This means that 1. computing is embedded in the physical world, 2. users can control computation through bodily movement and by interacting with physical input devices (which are not just another remote control), 3. the computers' output is integrated into the everyday world, e.g. through graphic projections or sound, 4. and with this computing moves into new social contexts, from schools over hospitals to care homes and wastewater plants.

Computing enters the built environment with large displays and sensors embedded into floors and doorways; it has become prominent in museums in the form of interactive installations that provide visitors with new means to experience and explore the topics of exhibitions; it has become an intrinsic part of everyday appliances, even being integrated into everyday furniture (think of the announcement of Microsoft's new Surface table); and has moved into schools, teachers experimenting with electronic whiteboards and Lego Mindstorm Robots.

Tangible Interaction is part of this larger trend, which emphasizes the priority of the physical or everyday world, aiming to retain human situatedness in the everyday world (inspired from Mark Weiser's vision of Ubiquitous Computing). The real world is not replaced, being seen as inferior, but is augmented with digital capabilities, retaining its role as our habitat.

Using tangible interaction as a design approach prioritizes tangibility, materiality, bodily interaction, and embeddedness in real spaces as principles of design. One of the core ideas that has driven this area is to quite literally allow users to grasp data with their hands, representing digital data objects and interaction tools with physical items (of which there are a great variety, including models, handles, and so on). People tend to react with immediate interest, even fascination, on encountering such systems, as tangibles seem to provide a counterbalance to the increasing virtualisation of our everyday world, re-introducing the sensuality of the haptic, and providing an opportunity for whole-body interaction. While this immediate appeal is evident, it is less clear what may be the best application areas - too many of the examples in the literature only exist as demos and conceptual prototypes.

Yet moving from toy examples and ‘proofs of concepts' to real applications in real contexts requires the accumulation of design knowledge together with serious evaluation. To design good systems, we need to know what makes up the ‘user experience'. Understanding this requires interdisciplinary research, since it is influenced by the physical interface, the software inside the system, and the interactions between these (plus the context of use). The dividing line between the domains of computer scientists, mechatronics experts, and the concerns of product and industrial designers has become fuzzy, with household appliances containing chips, displays, and even actuators that make them move.
Scaling up from toy examples requires mastery of complex behaviour as well as the creation of rich interactions suitable for the context of use. All of this means that an interdisciplinary perspective is needed to understand tangible interaction and to create good products.



Tangible Interaction as a strategic umbrella term

What is tangible interaction? Using this term has served to provide an encompassing viewpoint, easing collaboration and emphasizing commonalities across disciplines.
In computing and human-computer interaction, this author's initial fields, the terms ‘tangible media' and ‘tangible user interface' (TUI) have been in use since the late 90s to describe systems that quite obviously share characteristics with work from product design, which intends to create rich physical interactions. Yet the definitions used to describe tangible user interfaces were too restrictive to apply to these related areas - they focussed on providing tangible ‘handles' (physical pointers connected with the items interacted with) to support manipulation of digital data, substituting graphical with tangible interfaces (data-centered view). In fact, many of the systems developed under the TUI label employ tangible figurines and tokens with visualized data floating around them, quite literally demonstrating that the tangible objects mainly serve as an entry point into a digital data space. Interpreted as a rhetorical device in the context of technology trends at that time, this definition marked tangible interfaces as an antithesis to graphical user interfaces and virtual reality. As a side effect this put the emphasis on the control of data, whereas an alternative interest might be in controlling the real world (e.g. a heating controller). Or we might want to create completely new interaction styles, having to start thinking about the interactions and actions involved before deciding upon a digital model or on a physical form. This view is centred on bodily movement and aims to support rich expression and to enhance physical skills (instead of replacing these with button pushing which is less versatile in terms of bodily skills).
From the arts world, a space-centered view can be identified, emphasizing the creation of interactive and reactive spaces where computing as well as tangible elements are means to an end rather than explicit aims. The bodily movement of the ‘spectator' (who becomes an actor and performer) comes to be an integral part of the art installation, sometimes relying on bodily self-perception (invisible to the outside spectator). These types of art installations are often characterised by involvement of the whole body of the spectator, turning it into the means of interaction.
Traditionally engineering and computing have seen the interface as distinct and secondary to the functionality incorporated in a system. The interface is placed between the software and the user, enabling access to its features. Replacing one interface with another only changes the surface while the basic interaction process stays the same. Using the term ‘tangible interaction' places the focus on the design of the interaction instead of the visible interface, enabling us to discuss qualities of the interaction, and to focus on what people actually do with the system.

This encompassing perspective emphasises tangibility and materiality, physical embodiment of data, bodily interaction, and the embedding of systems in real spaces and contexts. Systems may very well combine graphical with tangible elements - there is no inherent conflict between the two if we no longer treat tangibility as antithetical.

History and Precursors to Tangible Interaction

In 1993 a special issue of the Communications of the ACM with the pro-grammatic title "Back to the Real World" argued that both desktop computers and Virtual Reality estrange humans from their ‘natural envi-ronment' and require too much translation work between real and digital worlds. Instead, one should augment and enrich the real world with digi-tal functionality. Whereas Human-Computer Interaction so far had been dominated by cognitive science, viewing humans as ‘information proces-sors', now ideas from ethnography, situated cognition, and phenomenol-ogy became influential in the argumentation for Augmented Reality and Ubiquitous Computing: „humans are of and in the everyday world".

While the underlying ideas for Tangible User Interfaces had been dis-cussed in this special issue, it took a few years for these to evolve into an interface concept in its own right. In 1995, Fitzmaurice and co-authors introduced the concept of a ‘graspable interface' , using wooden blocks as graspable handles to manipulate digital objects. A block is anchored to an object on the monitor by placing it on top. Graspable handles in com-bination with functionally dedicated input tools were argued to distribute input in space instead of time, to support bimanual action and to reduce mediation. Ishii and his students presented a more encompassing vision of ‘Tangible Bits' in 1997. The idea was to turn the whole world into an interface by connecting objects or surfaces with digital data and to present information with ‘ambient displays'. Emphasis lay on creating rich multi-sensory experiences that honour human dexterity. Ishii's work focused on using tangible objects to both manipulate and represent digi-tal content. While the graspable blocks were frequently reconnected and thus were abstract, iconic appearance of tangibles was now emphasized.
For most of the following decade research focused on system develop-ment to explore technical options. In recent years, the proof-of-concept stadium has led on to a more mature phase with more emphasis on con-ceptual design, user and field testing, critical reflection, theory and de-sign knowledge building. Connections with related developments in the design disciplines became stronger, motivating the recent move towards a broader understanding of the field as ‘Tangible Interaction'.
Several less well known precursors to the work of Ishii and his students have influenced the field. These addressed issues in specific application areas such as architecture, product design, and learning systems research; the ideas being picked up by HCI researchers as a new interface concept to be systematically pursued. Probably the first system that can be classi-fied as a tangible interface was Perlman's ‘Slot Machine', which uses physical cards representing language constructs to program the Logo Turtle. Sequences of cards could be inserted into slots of the machine, creating complex commandos such as ‘move forward twice'.
In the early 80s both Aish and Frazer were looking for alternatives to the then clunky and cumbersome CAD systems in architecture , wanting to enable the later inhabitants of buildings to partake in design discussions with architects, to make interaction with CAD more intuitive and to sup-port rapid idea testing. Both developed systems for ‘3D modelling' that consist of given blocks that users build a physical model from, which is scanned and interpreted by a computer. Users can configure the digital properties of blocks and let the computer calculate floor space, water pip-ing, or energy consumption. The simplest systems used an electronic breadboard grid that one would plug components onto. Frazer's ‘Intelli-gent Modelling Technique' was used in an experiment on CAD use by home self-builders. Residents in Lewisham Borough, London, built their homes following the so-called Segal approach using Frazer's construc-tion toolkit.
Often mentioned as inspiration for the notion of Tangible Interfaces are the works of product designer Durrell Bishop . He designed a ‘Marble Answering Machine' as a concept sketch at the Royal College of Art. In-coming calls are represented with coloured marbles that roll into a bowl. Placed into a second bowl, the messages are played back. Putting a mar-ble onto the phone calls the number from which the call originated. Bishop's designs rely on physical affordances and user's everyday knowledge. Most striking is how his works assign new meanings to ob-jects (object mapping), turning them into pointers to something else, into containers for data and references to other objects in a network. Many of his designs furthermore employ spatial mappings, deriving meaning from the context of an action (e.g. its place). A design sketch has a radio mounted in front of a poster. To search for a channel one needs to move the radio on its railings. What Bishop succeeded with in his designs, us-ing known objects as legible references for the aesthetics of new elec-tronic projects while refraining from simplistic literal metaphors, play-fully recombining meanings and actions, has remained a challenge and inspiration up to today.

Studying social interaction around tangible environments

One area where tangible interaction has clear advantages is in the support of social interaction and collaboration. This therefore has been the focus of the author's research in recent years. A central theme of this research is the interweaving of the material and social aspects of interaction.
The user experience with hybrid environments (coupling physical and digital elements) is influenced both by the physical elements (physical space, physical objects and their configuration) and the digital functionality. Together, these influence social interaction patterns, making certain behaviours more likely by inviting or demanding them, and discouraging other behaviours. The size of the clavier (a spatial audio installation) affords and requires simultaneous users to create an interesting soundscape, more or less asking for collaboration. Specific setups and types of interactive installations in a museum attracted particular use patterns, these being used mostly by individuals, pairs, or were surrounded by small groups. Tangible manipulation, that is the movement of objects or full-body movements, tends to be visible for others, who can learn from observing, and participate vicariously as bystanders (by e.g. commenting on the action), or decide to take an active part, waiting for an appropriate moment to join. Tangible interaction, by virtue of being visible, thus becomes performative - the dance on the clavier is unnecessary from a purely technical perspective!
The physical environment in which the system is embedded influences how people interpret what they encounter - it sets the stage. For example in a technical museum visitors know that there will be some exhibits which are designed to be hands-on, whereas in an art museum they hesitate to touch anything. When evaluating a museum exhibition with a range of interactive installations it was amazing how visitors tended to distinguish hands-on exhibits and traditionally displayed objects from afar, walking straight up to the former. The surroundings contribute meaning and atmosphere. One of the tricky issues for design is whether to support, to augment, or to contravene what is already there. The sound of the clavier and the lights had been chosen to augment the nightly atmosphere of a public garden, adding mystery and wonder. On getting to know the location of Chawton House, a historic estate in South England, the research team decided to direct visitors' attention towards taking notice of their surroundings, inspiring their imagination - all with as little information-push as possible.


Tangible interaction design is already happening nowadays in areas that we are not used to thinking of in terms of hi-tech development: museum installations, toys, remote controls, kitchen and entertainment devices... the list goes on. Digitally enhanced environments and intelligent objects are becoming a reality, supported by rapid technical advances. Too often, these are governed by techno-centric visions, designs are oriented by design patterns from desktop computing (buttons or drag and drop) or traditional design thinking that emphasises visuals and form over behaviour and interaction. An encompassing perspective on tangible interaction can help us to shape this development, making use of the new options while retaining the intimate connections of we as humans with our own bodies and the physical environment.