Littérature scientifique sur le sujet « Multimodal user interface »

This research work is meant to offer a theoretical contribution to the study of Visual Design: in particular, the concept of Multimodal Directorship in New Media is approached, defined and developed. Multimodal Directorship is presented as knowledge framewok for research and experimentation of communication languages supporting human-computer interfaces design. The research about Multimodal Directorship gives a theoretical ground to develop better hermeneutic and design methods. The starting point of this research is to define New-Media as digital communication tools structured around hypertextual links. Many, active communication channels (i. e. visual, textual, acoustic and so on) convey simoultaneously information to the user perception. Every single channel does not act indipendently: it must work as a co-operating element with the other modes of communication within a complex system. Interface design in New Media is part of an interdisciplinary context where a system of direction is identified, articulating the design of modes and languages specific of different perceptions into a global, single, efficient medium. As stated in the cognitive psychology approach of the San Diego School, and from the metaphore of the multimedial authoring software of Canter, the human-computer interaction is very similar to a theatre drama. In this framework the action of the designer is analysed and accordingly the metaphore is resolved. Following this approach the communications designer acts very similarly to a movie or theater director coordinating all the psycho-perceptual effects. The knowledge and the know-how extend therefore outside the traditional field of composition and graphical communication and the designer is involved in a global approach to project. The designer is the privileged author of the intertextual script that writes the different expression modes and the user, as co-author, during the interaction will activate the different communication modes by experiencing the interface. Designers tried to adapt in a mimetical and naïve way the previouse experiences usual in professional practice to digital media. The potential expressions of new-media have been missed and misused. Only the mymesis of traditional media (i. e. like printed paper) was exploited. The challenge for the Multimodal Directorship is to find out a new grammatic and sintaxis to sintetyze the different communication modalities. The methodology doesn’t depend on an episodic technological development aiming to the creation of new languages to cope with the technological innovation. The directorship approach should develop within an abstract, conceptual framework; is constantly growing through interaction with other research fields like semiotic, cognitive psycology, information technology and so on. The knowledge that comes out from design research modifies the professional process. The Multimodal Directorship as discipline of new-media design utilies and reshapes knowledge previously produced within its own field, and foster constantly a critical knowledge on the professional practice.

Abstract Usability methodology has matured into a well-defined, industrially relevant field with its own findings, theories, and tools, with roots in applying information technology to user interfaces ranging from control rooms to computers, and more recently to mobile communications devices. The purpose is regularly to find out the users’ goals and to test whether a design fulfils the usability criteria. Properly applied, usability methods provide reliable and repeatable results, and are excellent tools in fine-tuning existing solutions The challenges of usability methodologies are in finding new concepts and predicting their characteristics before testing, especially when it comes to the relatively young field of mobile user interfaces. Current usability methods concentrate on utilising available user-interface technologies. They do not provide means to clearly identify, e.g., the potential of auditory or haptic output, or gestural input. Consequently, these new interaction techniques are rarely used, and the long-envisioned useful multimodal user interfaces are yet to appear, despite their assumed and existing potential in mobile devices. Even the most advocated and well-known multimodal interaction concepts, such as combined manual pointing and natural language processing, have not materialised in applications. An apparent problem is the lack of a way to utilise a usage environment analysis in finding out user requirements that could be met with multimodal user interfaces. To harness the full potential of multimodality, tools to identify both little or unused and overloaded modalities in current usage contexts are needed. Such tools would also help in putting possibly existing novel interaction paradigms in context and pointing out possible deficiencies in them. In this thesis, a novel framework for analysing the usage environment from a user-centric perspective is presented. Based on the findings, a designer can decide between offering a set of multiple devices utilising various user-interface modalities, or a single device that offers relevant modalities, perhaps by adapting to the usage environment. Furthermore, new techniques for creating mobile user interfaces utilising various modalities are proposed. The framework has evolved from the experiences gathered from the designs of experimental and actual product-level uni- and multimodal user interface solutions for mobile devices. It has generated novel multimodal interaction and interface techniques that can be used as building blocks in system designs.