Dissertations / Theses on the topic 'Auditory display'

This thesis explores how audio can be better incorporated into how people access information and does so by developing approaches for creating three-dimensional audio environments with low processing demands. This is done by investigating three research questions. Mobile applications have processor and memory requirements that restrict the number of concurrent static or moving sound sources that can be rendered with binaural audio. Is there a more e cient approach that is as perceptually accurate as the traditional method? This thesis concludes that virtual Ambisonics is an ef cient and accurate means to render a binaural auditory display consisting of noise signals placed on the horizontal plane without head tracking. Virtual Ambisonics is then more e cient than convolution of HRTFs if more than two sound sources are concurrently rendered or if movement of the sources or head tracking is implemented. Complex acoustics models require signi cant amounts of memory and processing. If the memory and processor loads for a model are too large for a particular device, that model cannot be interactive in real-time. What steps can be taken to allow a complex room model to be interactive by using less memory and decreasing the computational load? This thesis presents a new reverberation model based on hybrid reverberation which uses a collection of B-format IRs. A new metric for determining the mixing time of a room is developed and interpolation between early re ections is investigated. Though hybrid reverberation typically uses a recursive lter such as a FDN for the late reverberation, an average late reverberation tail is instead synthesised for convolution reverberation. Commercial interfaces for music search and discovery use little aural information even though the information being sought is audio. How can audio be used in interfaces for music search and discovery? This thesis looks at 20 interfaces and determines that several themes emerge from past interfaces. These include using a two or three-dimensional space to explore a music collection, allowing concurrent playback of multiple sources, and tools such as auras to control how much information is presented. A new interface, the amblr, is developed because virtual two-dimensional spaces populated by music have been a common approach, but not yet a perfected one. The amblr is also interpreted as an art installation which was visited by approximately 1000 people over 5 days. The installation maps the virtual space created by the amblr to a physical space.

Thesis: S.M., Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2016.
Cataloged from PDF version of thesis.
Includes bibliographical references (pages 112-118).
Two projects in scientific data sonification are presented. 'Quantizer' is a platform that enables composers to develop artistic sonification schemes using real-time data from the ATLAS detector at CERN. Three sample audio streams are available for real-time consumption by the public and the public engagement potential for the project is studied. 'Rotator' uses sonification as a practical tool for analysis of high dimensional data Users can swipe data between their auditory and visual channels in order to best perceive the structure of a dataset. A dual audio-visual presentation mode is found to be a promising alternative to use of a purely visual display mode.
by Juliana Mae Cherston.
S.M.

This thesis draws from music technology to create novel sonifications of heart rate information that may be of clinical utility to physicians. Current visually-based methods of analysis involve filtering the data, so that by definition some aspects are illuminated at the expense of others, which are decimated. However, earlier research has demonstrated the suitability of the auditory system for following multiple streams of information. With this in mind, sonification may offer a means to display a potentially unlimited number of signal processing operations simultaneously, allowing correlations among various analytical techniques to be observed. This study proposes a flexible listening environment in which a cardiologist or researcher may adjust the rate of playback and relative levels of several parallel sonifications that represent different processing operations. Each sonification "track" is meant to remain perceptually segregated so that the listener may create an optimal audio mix. A distinction is made between parameters that are suited for illustrating information and parameters that carry less perceptual weight, which are employed as stream separators. The proposed sonification model is assessed with a perception test in which participants are asked to identify four different cardiological conditions by auditory and visual displays. The results show a higher degree of accuracy in the identification of obstructive sleep apnea by the auditory displays than by visual displays. The sonification model is then fine-tuned to reflect unambiguously the oscillatory characteristics of sleep apnea that may not be evident from a visual representation. Since the identification of sleep apnea through the heart rate is a current priority in cardiology, it is thus feasible that sonification could become a valuable component in apnea diagnosis.

This thesis investigates the design of audio for feedback in human-technology interaction— auditory displays. Despite promising progress in research and the potential benefits, we currently see little impact of audio in everyday interfaces. Changing interaction paradigms, new contexts of use and inclusive design principles, however, increase the need for an efficient, non-visual means of conveying information. Motivated by these needs, this work describes the development and evaluation of a methodological design framework, aiming to enhance knowledge and skill transfer in auditory display design and to enable designers to build more efficient and compelling auditory solutions. The work starts by investigating the current practice in designing audio in the user interface. A survey amongst practitioners and researchers in the field and a literature study of research papers highlighted the need for a structured design approach. Building on these results, paco – pattern design in the context space has been developed, a framework providing methods to capture, apply and refine design knowledge through design patterns. A key element of paco, the context space, serves as the organising principle for patterns, artefacts and design problems and supports designers in conceptualising the design space. The evaluation of paco is the first comparative study of a design methodology in this area. Experts in auditory display design and novice designers participated in a series of experiments to determine the usefulness of the framework. The evaluation demonstrated that paco facilitates the transfer of design knowledge and skill between experts and novices as well as promoting reflection and recording of design rationale. Alongside these principle achievements, important insights have been gained about the design process which lay the foundations for future research into this subject area. This work contributes to the field of auditory display as it reflects on the current practice and proposes a means of supporting designers to communicate, reason about and build on each other’s work more efficiently. The broader field of human-computer interaction may also benefit from the availability of design guidance for exploiting the auditory modality to answer the challenges of future interaction design. Finally, with paco a generic methodology in the field of design patterns was proposed, potentially similarly beneficial to other designing disciplines.

Thesis (Ph. D.)--University of North Carolina at Chapel Hill, 2008.
Title from electronic title page (viewed Dec. 11, 2008). "... in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Computer Science." Discipline: Computer Science; Department/School: Computer Science.

This study investigated automobile drivers' ability to localize auditory and haptic (touch) alarms in terms of speed and accuracy. Thirty-two subjects, balanced across age (20-30 years old and 60-70 years old) and gender, participated in the study. Subjects were screened for minimum hearing of 40 dB for 500 Hz through 4000 Hz auditory tones, and maximum bilateral hearing differences of 10 dB. The experiment consisted of subjects identifying the target location of an alarm while driving a 2001 Buick LeSabre at 55 mph in light traffic. Four alarm modes were tested: 1) an auditory broadband alarm, 2) a haptic seat, 3) a combination of the haptic and the auditory alarm modes, and 4) a combination of the haptic alarm mode with a non-directional auditory alarm played from the front speakers of the vehicle. The alarms were evoked from eight target locations: the front-left, front, front-right, right, back-right, back, back-left, and left. The target locations of the auditory alarm mode existed around the interior of the car cabin using the vehicle's stock sound system speakers. The haptic alarm target locations existed in the bottom of the driver seat using an eight-by-eight grid of actuators. The experimenter evoked the alarms while subjects drove along a two-lane highway, and the alarms were not associated with any actual collision threat. Subjects were instructed to quickly identify the location of the alarm by calling them out, while being as correct as possible. Their choice response time and target location selection was recorded. The alarms were presented approximately every minute during fifteen-minute intervals over the duration of two and a half hours. Subjects completed questionnaires regarding their preference to the alarm modes. Under the conditions investigated, subjects localized the haptic alarm mode faster and more accurately than the auditory alarm mode. Subjects performed equally well with the haptic alarm mode and the two auditory and haptic combination alarm modes in terms of speed and accuracy in identifying their location. Subjects did express a preference for the addition of the auditory component to the haptic alarm mode, perhaps owing to a heightened sense of urgency. However, subjects preferred the haptic alarm mode on its own in response to hypothetical false alarm questions, perhaps because it was less annoying. Alarm mode discriminability was believed to affect localization accuracy and response time owing to its effect on the likelihood of correctly identifying a target location and the attention resources required to differentiate adjacent target locations.
Master of Science

The arrival of inexpensive high fidelity audio capabilities for personal computers has been accompanied by sweeping claims of enhanced educational effectiveness. Research has only served to confuse these claims with both significant and non-significant findings. This may be due partly to the single dimensional model which has been assumed when examining audio. When computer-based audio is examined as a multi-dimensional display analogous in capability and function to the more familiar visual display (computer screen), a different picture emerges. Although it is now clear that integrated audio can enhance learning, the instructional designer is still left with little guidance whether or how to use audio to best effect. This study has examined whether there are, and if so which communications filling specific educational functions (instructional text and aural examples) can be delivered through different output channels (visual display channel [computer screen] and for auditory display channel [earphones]) to best effect. Randomly selected II th and 12th grade students from the same high school were given a computer-based instructional sequence with aural content. The treatment was varied to conform to a two-way factorial design examining both instructional narrative and aural example when offered via (a) the visual display only, (b) the auditory display only, and (c) simultaneously in both displays. All treatments were followed by an on-line, simulation-based, problem-solving examination. Outcome scores, response latency and elapsed treatment times were analyzed with two-way analysis of variance and multiple comparison procedures to identify any significant main or interaction effects. The independence of communications serving specific educational functions was confirmed. Instructional narrative was found to be more effective (p < .0 I) when presented as part of the auditory display either alone or in concert with the visual display. Aural examples were most effectively (p < .05) displayed simultaneously as part of the auditory and visual displays. Time required to complete instruction was not affected by the addition of an auditory display. Lack of an auditory display can be remediated by exposure to and additional practice with the targeted aural content.

The prospect of computer applications making "noises" is disconcerting to some. Yet the soundscape of the real world does not usually bother us. Perhaps we only notice a nuisance? This thesis is an approach for designing sounds that are useful information rather than distracting "noise". The approach is called TaDa because the sounds are designed to be useful in a Task and true to the Data. ¶ Previous researchers in auditory display have identified issues that need to be addressed for the field to progress. The TaDa approach is an integrated approach that addresses an array of these issues through a multifaceted system of methods drawn from HCI, visualisation, graphic design and sound design. A task-analysis addresses the issue of usefulness. A data characterisation addresses perceptual faithfulness. A case-based method provides semantic linkage to the application domain. A rule-based method addresses psychoacoustic control. A perceptually linearised sound space allows transportable auditory specifications. Most of these methods have not been used to design auditory displays before, and each has been specially adapted for this design domain. ¶ The TaDa methods have been built into computer-aided design tools that can assist the design of a more effective display, and may allow less than experienced designers to make effective use of sounds. The case-based method is supported by a database of examples that can be searched by an information analysis of the design scenario. The rule-based method is supported by a direct manipulation interface which shows the available sound gamut of an audio device as a 3D coloured object that can be sliced and picked with the mouse. These computer-aided tools are the first of their kind to be developed in auditory display. ¶ The approach, methods and tools are demonstrated in scenarios from the domains of mining exploration, resource monitoring and climatology. These practical applications show that sounds can be useful in a wide variety of information processing activities which have not been explored before. The sounds provide information that is difficult to obtain visually, and improve the directness of interactions by providing additional affordances.

Recent guidelines recommend oxygen saturation (SpO(2)) levels of 90%-95% for preterm neonates on supplemental oxygen but it is difficult to discern such levels with current pulse oximetry sonifications. We tested (1) whether adding levels of tremolo to a conventional log-linear pulse oximetry sonification would improve identification of SpO(2) ranges, and (2) whether adding a beacon reference tone to conventional pulse oximetry confuses listeners about the direction of change. Participants using the Tremolo (94%) or Beacon (81%) sonifications identified SpO(2) range significantly more accurately than participants using the LogLinear sonification (52%). The Beaton sonification did not confuse participants about direction of change. The Tremolo sonification may have advantages over the Beacon sonification for monitoring SpO(2) of preterm neonates, but both must be further tested with clinicians in clinically representative scenarios, and with different levels of ambient noise and distractions. Crown Copyright (C) 2016 Published by Elsevier Ltd. All rights reserved.

Human auditory perception is suited to receiving and interpreting information from the environment but this knowledge has not been used extensively in designing computer-based information exploration tools. It is not known which aspects of sound are useful for accurately conveying information in an auditory display. An auditory display was created using PD, a graphical programming language used primarily to manipulate digital sound. The interface for the auditory display was a blank window. When the cursor is moved around in this window, the sound generated would changed based on the underlying data value at any given point. An experiment was conducted to determine which attribute of sound most accurately represents data values in an auditory display. The four attributes of sound tested were frequency-sine waveform, frequency-sawtooth waveform, loudness and tempo. 24 subjects were given the task of finding the highest data point using sound alone using each of the four sound treatments. Three dependent variables were measured: distance accuracy, numeric accuracy, and time on task. Repeated measures ANOVA procedures conducted on these variables did not rise to the level of statistical significance (α=.05). None of the sound treatments was more accurate than the other as representing the underlying data values. 52% of the trials were accurate within 50 pixels of the highest data point (target). An interesting finding was the tendency for the frequency-sin waveform to be used in the least accurate trial attempts (38%). Loudness, on the other hand, accounted for very few (12.5%) of the least accurate trial attempts. In completing the experimental task, four different search techniques were employed by the subjects: perimeter, parallel sweep, sector, and quadrant. The perimeter technique was the most commonly used.

When using non-speech as an information delivery medium, information patterns are mapped onto auditory variables. The efficiency of delivering information to listeners depends on the usage of auditory variables. A discriminable auditory display can reduce the ambiguity of listeners’ perceptual representation of auditory inputs. However, there is lacking an understanding of the way in which listeners decode information embedded in sound through identifying auditory variations. This thesis aims to develop an understanding of the influence of auditory variables on the identifiability of the features of auditory display. The auditory display in this thesis relates to auditory graph where series of quantitative values were mapped onto pitches and presented over time. This thesis explores the effects of spatial separation and timbre on the identifiability of the features of simultaneously presented auditory graphs. Features that listeners were required to identify included an increased periodic modulation in pitch that occurred in only one stream of two displayed auditory streams or a difference in the overall shape of the pitch contour of two displayed auditory streams. Both of these tasks required divided attention between stimuli forming two auditory streams, which were presented via headphones in order either to appear at two separate spatial locations, or to be co-located at a single location. Also, the displayed stimuli could differ in timbre, allowing for an evaluation of the relative influences of timbral difference and spatial separation on the identifiability of the target features. One hypothesis tested here was that when simultaneously presented auditory graphs were composed of the same timbre, their pitch contours would be more easily identified when displayed at spatially separated locations. A second hypothesis tested here was that the enhancement of identification performance due to spatial separation of simultaneously presented auditory graphs would be greatest for auditory streams that differ least in timbre, and that discrimination performance given streams with more distinct timbre would be less influenced by spatial separation of the streams. In a subsequent divided attention study using only spatially separated auditory streams, the perceptual salience of timbral differences in an identification task was investigated using timbral attribute ratings. Finally, in a data sonification case study employing multiple auditory streams exhibiting spatial and timbral differences, a qualitative approach was taken that aimed to deepen understanding of the way people listen and think in such contexts. Overall, the applied method demonstrated the influence of spatial separation and timbral differences on the feature identification for simultaneously presented auditory streams. Although there is potential for further refinement of the methods applied here, this work provided an examination and elucidation of the relative salience of two prominent factors involved in determining the effectiveness of concurrent auditory displays and delineated some of their limitations in supporting two different identification tasks. The responses, collected using an open questionnaire administered after participation in some of the experiments, highlighted the intricate relationships that exist in the allocation of attention between perceptually segregated concurrent streams, and in the auditory display processing activities and listening strategies of human subjects engaged in divided attention tasks.

This thesis presents the development and evaluation of a new approach (Sound- TrAD) to designing auditory interfaces. The proposed approach combines practices and concepts from film soundtrack composition with established approaches to general user interface design. The synthesis of the two design approaches from different areas of design into a novel approach may be viewed as an example of conceptual integration, (also known as conceptual blending). The process of developing and evaluating SoundTrAD broadly follows a methodology of Research through Design. The thesis presents four user studies as part of an iterative design and evaluation process. Each study involves a mixture of expert and novice end-users which provides new information and identifi es new questions and design issues for the subsequent studies. The fi rst study explores how an idea from fim composition (the cue sheet) can be used in auditory interface design to help designers place and organise sound elements, and to better understand auditory design spaces. In order to make this concept work in the new context, it is combined with the scenario concept from general interaction design to provide designers with reference linear sequences of events and actions. The second study used thematic analysis to investigate how information to be sonifed can be characterised and analysed for features that can be mapped in to sound. The study also explores the development of a timeline on which the sound design ideas from soundtrack composition for individual events, can be placed and in principle moved in order to cater for multiple use-case scenarios. The third study presents an iteration of this, including further development of both the task analysis and mapping technique. The study also explores the idea in principle of an interactive timeline that can be manipulated by the designer in order to re-arrange and audition sound events. The final study brings the studies together by obtaining feedback on the success of a nal version of SoundTrAD.

This simulator-based study examined the use of conventional auditory warnings (tonal, non-verbal sounds) and auditory icons> (representational non-verbal sounds), alone and in combination with a dash-mounted visual display, to present information about impending collision situations to commercial motor vehicle operators. Brake response times were measured for impending front-to-rear collision scenarios under six display configurations, two vehicle speeds, and two levels of headway. Accident occurrence was measured for impending side collision scenarios under two vehicle speeds, two levels of visual workload, two auditory displays, absence/presence of mirrors, and absence/presence of dash-mounted iconic display. Subjective preference data was also obtained from participants. For both front-to-rear and side collision scenarios, auditory icons elicited significantly improved driver performance over conventional auditory warnings. Driver performance improved when collision warning information was presented through multiple modalities. Brake response times were significantly faster for impending front-to-rear collision scenarios using the longer headway criterion. The presence of mirrors significantly reduced the number of accidents for impending side collision scenarios. Subjective preference data indicated that participants preferred multi-modal displays over single-modality displays. A technique for systematically identifying, selecting, and evaluating candidate auditory icons was also developed. The potential exists to expand upon these developments, toward the goals of identifying appropriate auditory icons, improving operator performance, and developing information display techniques to effectively managing workload across multiple modalities.
Master of Science

Många systemutvecklare idag saknar kunskap om hur ljud kan användas i system för att öka användarupplevelsen och är skeptiska till ljudets användningspotential. Auditory display är ett samlingsnamn för olika tekniker som nyttjar ljudet som ett medium för att kommunicera olika typer av data och information från systemet till användaren. Då mycket av forskningen rörande auditory display fokuserar på utvecklingen av hjälpmedel för synskadade istället för mer generella system för användare utan särskilda behov ser vi att det finns ett behov av en litteraturstudie med fokus på den sistnämnda gruppen system. Vi ställde frågan "hur kan auditory display tillämpas vid utveckling av traditionella IT-system" och genomförde en litteraturstudie där 23 artiklar analyserade för att identifiera olika tillämpningsområden för auditory display, med syfte att återge dessa tillämpningsområden i ett format som är av intresse för utvecklare som vill kunna använda ljud i sina system men inte vet vart de ska börja. Resultaten visar att auditory display kan användas till god effekt exempelvis vid övervakning av nätverkstrafik, i gränssnitt och widgets och i fordonsgränssnitt i bilar. Vi föreslår även lovande användningsområden som bör undersökas vidare av framtida forskare.
Today, there is a certain lack of knowledge on how sound can be utilized in systems to enhance the user experience among systems developers and many developers have a skeptical outlook on the usability of sound. Auditory display is an umbrella term for an array of different techniques that utilize sound as a medium to communicate different sorts of information and data from the system to the user. As much of the research revolving around auditory display has the development of accessibility tools for visually impaired as its sole focus, instead of more general systems intended for users without specific needs, we see that there is a need for a literature review focused on the latter. We asked ourselves the question "how can auditory display be utilized in the development of traditional IT-systems?" and carried out a literature review where 23 articles were analyzed to identify different use cases, with the purpose of presenting these use casers in a way that can be used by developers who are interested in implementing sound in their systems but are unsure where to start. Results indicate that auditory display can be used to good effect in, among others, systems for monitoring network traffic, user interfaces and widgets and in-vehicle technologies. Lastly, we propose promising potential use cases that are in need of further research.

Virtual auditory display (VAD) systems rely upon binaural technology to render sound sources at controlled directions in virtual acoustic spaces. The accuracy and precision with which human listeners can localise those sound sources, particularly in terms of perceived source elevation, depends upon spectral variation in the incident sound that is due to its interaction with head related transfer functions (HRTFs). The spectral processing developed in the current thesis has been optimised to reduce listener uncertainty regarding the perceived elevation of virtual sound sources, and to improve the overall spatial perception. A subset of individually measured HRTFs that supported the highest localisation accuracy was identified via preliminary listening sessions and used in the formation of a single ‘Collective’ HRTFs dataset that could be deployed for the entire group of listeners in a customised fashion. The customisation employed individually determined frequency scaling that was applied to the selected HRTFs before deploying the Collective HRTFs dataset, which could be readjusted for each individual through a calibration procedure that was based upon the individual’s localisation judgments. An evaluation of this customised HRTF dataset in a spatial auditory display of ecologically valid sound sources demonstrated improvement in localisation performance, in comparison with both the accuracy and precision of results obtained using individually measured HRTFs. Furthermore, the results informed the development of an adaptive processing of the proximal sound sources at runtime that showed good potential for improving localisation performance in a manner that adapts to listener responses. Based upon a runtime analysis of input sound source spectral variation, the adaptive processing was designed to improve accuracy and reduce uncertainty in apparent source elevation angle for the listener, and thus improve overall localisation performance.

Vehicles are becoming increasingly autonomous, as automotive industries are investing in innovative technology. Therefore the technology becomes more available and affordable, making it possible for Toyota Material Handling Europe (TMHE) to introduce automated features in their trucks. Vehicles that have a forward collision warning system, and thus are partly autonomous, are involved in less accidents than those without. In manufacturing industries there is currently a problem with truck collisions and an automated solution might be a suitable way to prevent these. When implementing an automation device, human machine interaction and user-friendliness are aspects to keep in mind during the development. The thesis concerns how autonomous features can assist the truck driver, and how to provide the driver with intuitive feedback. The purpose was to ensure the drivers’ and surrounding personnel’s safety as well as increase the productivity. Research was performed regarding in what situation an assisting device is needed and how to communicate information in an intuitive manner to help the driver in this situation. A conceptual interface was developed that allows communication between the driver and a future all-knowing system, that tracks all objects and personnel in a warehouse. The drivers have had a central role in the process. The observations were performed in the TMHE warehouse to identify situations. The most perilous and frequent situation was when drivers need to focus both in the fork and drive wheel directions simultaneously. This either puts the surroundings or the driver in danger. A conceptual interface was developed to help the driver in this situation. This resulted in a concept implementable in both current and future trucks, to harmonise the solution and ensure a safe warehouse environment. A lo-fi prototype was constructed and evaluated iteratively with drivers to ensure the quality and usability of the concept. The resulting feedback solution consists of sounds from speakers mounted in the headrest and a display interface with warning symbols. The sounds are directional to notify the driver if the danger is to the left or right behind his back. If the danger is only semi-close, the driver receives a warning, but if it is very close, the truck is stopped autonomously. The symbols appear on the display simultaneously as the sounds are heard, to provide further feedback. Additionally, an Autonomous Positioning feature has been developed, that consists of symbols and buttons on the display interface, as well as an alert sound from the display to indicate the system’s activation and deactivation. Safety is enhanced since neither personnel nor trucks are in risk of collision when implementing the concept. As the concept helps the driver position the truck effortlessly towards the pallet the productivity is also improved.

Visual workload demand within the cockpit is reaching saturation, whereas the haptic sense (proprioceptive and tactile sensation) is relatively untapped, despite studies suggesting the benefits of haptic displays. MRT suggests that inputs from haptic displays will not interfere with inputs from visual or auditory displays. MRT is based on the premise that multisensory integration occurs only after unisensory processing. However, recent neuroscientific findings suggest that the distinction between unisensory versus multisensory processing is much more blurred than previously thought. This programme of work had the following two research objectives: 1. To examine whether multiple haptic inputs can be processed at the same time without performance decrement - Study One 2. To examine whether haptic inputs can be processed at the same time as visual or auditory inputs without performance decrement - Study Two In Study One participants performed dual-tasks, consisting of same-sense tasks (tactile or proprioceptive) or different-sense tasks (tactile and proprioceptive). These tasks also varied in terms of processing code, in line with MRT. The results found significantly more performance decrement for the same-sense dual-tasks than for the different-sense dual-tasks, in accordance with MRT, suggesting that performance will suffer if two haptic displays of the same type are used concurrently. An adjustment to the MRT model is suggested to incorporate these results. In Study Two, participants performed different-sense dual-tasks, consisting of auditory or visual tasks with tactile or proprioceptive tasks. The tasks also varied in terms of processing code. Contrary to MRT, the results found that when processing code was different, there was significant performance decrement for all of the dual-tasks, but not when processing code was the same. These results reveal an exception to two key MRT rules, the sensory resource rule and the processing code rule. It is suggested that MRT may be oversimplistic and other factors highlighted by recent neuroscientific research should be taken into account in theories of dual-task performance.

Visual workload demand within the cockpit is reaching saturation, whereas the haptic sense (proprioceptive and tactile sensation) is relatively untapped, despite studies suggesting the benefits of haptic displays. MRT suggests that inputs from haptic displays will not interfere with inputs from visual or auditory displays. MRT is based on the premise that multisensory integration occurs only after unisensory processing. However, recent neuroscientific findings suggest that the distinction between unisensory versus multisensory processing is much more blurred than previously thought. This programme of work had the following two research objectives: 1. To examine whether multiple haptic inputs can be processed at the same time without performance decrement - Study One 2. To examine whether haptic inputs can be processed at the same time as visual or auditory inputs without performance decrement - Study Two In Study One participants performed dual-tasks, consisting of same-sense tasks (tactile or proprioceptive) or different-sense tasks (tactile and proprioceptive). These tasks also varied in terms of processing code, in line with MRT. The results found significantly more performance decrement for the same-sense dual-tasks than for the different-sense dual-tasks, in accordance with MRT, suggesting that performance will suffer if two haptic displays of the same type are used concurrently. An adjustment to the MRT model is suggested to incorporate these results. In Study Two, participants performed different-sense dual-tasks, consisting of auditory or visual tasks with tactile or proprioceptive tasks. The tasks also varied in terms of processing code. Contrary to MRT, the results found that when processing code was different, there was significant performance decrement for all of the dual-tasks, but not when processing code was the same. These results reveal an exception to two key MRT rules, the sensory resource rule and the processing code rule. It is suggested that MRT may be oversimplistic and other factors highlighted by recent neuroscientific research should be taken into account in theories of dual-task performance.

Thesis (S.M.)--Massachusetts Institute of Technology, Dept. of Aeronautics and Astronautics, 2006.
Includes bibliographical references (p. 55-56).
An air-combat pilot must both control the aircraft and locate targets. The objective of this study was to determine combined effects of using the Non-Distributed Flight Reference and 3-D audio/visual cueing system for the purpose of easing pilot control and aiding target location. The Non-Distributed Flight Reference was compared against the Military Standard display, while three different target cues were presented for target acquisition: (1) 3-D Audio Only, a voiced signal = "Target-Target", (2) Visual Only, a line emanating from the aim-sight reticle (located in the center of the subject's field of view) pointing towards the target aircraft, and (3) a Combined Audio/Visual cue, providing both cues. The subjects were asked to fly a simulated aircraft at a commanded heading and altitude while targets appeared from 15 different locations (three elevations 0⁰, +40⁰, and five azimuths 0⁰, ±30⁰, ±60⁰). Subject deviations from heading and altitude were used to determine flight performance, and target cue effectiveness was determined by recording the time to target. The results of this experiment showed that there was no significant difference (p < 0.05) in flight performance between the Non-Distributed Flight Reference and the Military Standard displays. The combined Audio/Visual Cueing system resulted in the fastest search times.
(cont.) These results are consistent with the findings from past research. The cleaner more integrated Non-Distributed Flight Reference allowed subjects to visually acquire targets quicker with no decrease in flight performance. Using the Non-Distributed Flight Reference display for attitude information combined with a 3-D Audio/Visual cueing system for target acquisition is a promising method of reducing search time during a visual search task.
by Carlos Pinedo.
S.M.

Physics engines have become increasingly prevalent in everyday technology. In the context of this thesis they are regarded as a readily available data set that has the potential to intuitively present the process of sonification to a wide audience. Unfortunately, this process is not the focus of attention when formative decisions are made concerning the continued development of these engines. This may reveal a missed opportunity when considering that the field of interactive sonification upholds the importance of physical causalities for the analysis of data through sound. The following investigation deliberates the contextual framework of this field to argue that the physics engine, as part of typical game engine architecture, is an appropriate foundation on which to design and implement a dynamic toolset for interactive sonification. The basis for this design is supported by a number of significant theories which suggest that the underlying data of a rigid body dynamics physics system can sustain an inherent audiovisual metaphor for interaction, interpretation and analysis. Furthermore, it is determined that this metaphor can be enhanced by the extraordinary potential of the computer in order to construct unique abstractions which build upon the many pertinent ideas and practices within the surrounding literature. These abstractions result in a mental model for the transformation of data to sound that has a number of advantages in contrast to a physical modelling approach while maintaining its same creative potential for instrument building, composition and live performance. Ambitions for both sonification and its creative potential are realised by several components which present the user with a range of options for interacting with this model. The implementation of these components effectuates a design that can be demonstrated to offer a unique interpretation of existing strategies as well as overcoming certain limitations of comparable work.

Negli ultimi anni, i progressi delle tecnologie di elaborazione digitale dei segnali hanno favorito l’utilizzo del suono all’interno dei sistemi multimediali, non soltanto in virtù delle sue caratteristiche musicali, ma anche come mezzo per rappresentare informazioni e dati più o meno complessi. Sfruttare l’udito in combinazione con la vista viene tanto naturale nella quotidianità che quasi non ci si accorge di quanto questo sistema sia complesso. Oggigiorno poter usufruire di questo strumento anche quando si interagisce con la tecnologia è cruciale. In questi termini la Sonificazione ha reso possibile porre nuovi orizzonti in moltissimi campi scientifici, attraverso l’efficacia e l’efficienza dimostrata nei vari casi di studio. L’obiettivo di questa tesi è di trasmettere al lettore una conoscenza del concetto di Sonificazione, soffermandosi in particolare sugli aspetti tecnici e sui vantaggi e gli svantaggi che essa può offrire. Oltre alle nozioni già presenti nello stato dell’arte, si sono voluti presentare due casi di studio recenti, nel tentativo di illustrare come questa disciplina possa avere un impatto fondamentale nella vita di tutti i giorni.

This thesis describes work from both the University of Cambridge in the lab of Berthold Hedwig and from the HHMI Janelia Research Campus in the lab of Michael Reiser. At the University of Cambridge, my work involved the development and demonstration of a method for electrophoretically delivering dyes and tracers for anatomical and functional imaging into animals that are not amenable to genetic labelling techniques. Using this method in locusts and crickets - model systems of particular interest for their acoustic communication - I successfully delivered polar fluorescent dyes and tracers through the sheath covering the auditory nerve, simultaneously staining both the peripheral sensory structures and the central axonal projections without destroying the nerve's function. I could label neurons which extend far from the tracer delivery site on the nerve as well as local neuron populations through the brain's surface. I used the same method to deliver calcium indicators into central neuropils for in vivo optical imaging of sound-evoked activity, as well as calling song-evoked activity in the brain. The work completed at the Janelia Research Campus began with the development of a modern version of a modular LED display and virtual reality control system to enable research on the visual control of complex behaviors in head-fixed animals. The primary advantages of our newly developed LED-based display over other display technologies are its high-speed operation, brightness uniformity and control, precise synchronization with analog inputs and outputs, and its ability to be configured into a variety of display geometries. Utilizing the system's fast display refresh rates, I conducted the first accurate characterization of the upper limits of the speed sensitivity of Drosophila for apparent motion during flight. I also developed a flexible approach to presenting optic flow scenes for functional imaging of motion-sensitive neurons. Finally, through the on-line analysis of behavioral measures, image rendering, and display streaming with low latency to multi-color (UV/Green) LED panels, I demonstrated the ability to create more naturalistic stimuli and interactive virtual visual landscapes. Lastly, I used this new visual display system to explore a newly discovered cell-type that had been implicated in higher-order motion processing from a large genetic screen of visually-guided behavior deficits. Using genetic silencing and activation methods, and by designing stimuli that modeled the optic flow encountered during different types of self-motion, colleagues in the Reiser lab and I showed that this cell-type - named Lobula Plate Columnar 1 (LPC1) - is required for the stopping behavior of walking flies caused by back-to-front translation motion but is not involved in the rotational optomotor response. Using calcium imaging, I found that LPC1 was selectively excited by back-to-front motion on the eye ipsilateral to the neuron population and inhibited by front-to-back motion on the contralateral eye, demonstrating a simple mechanism for its selectivity to translation over rotation. I also examined an anatomically similar cell type - named Lobula-Lobula Plate Columnar type 1 (LLPC1) - and found that its selectivity results from a similar but opposite calculation for the detection of front-to-back translational motion. The detection of back-to-front motion had previously been hypothesized to be useful for collision avoidance, and this work provides a neural mechanism for how this detection could be accomplished, as well as providing a platform from which to explore the larger network for translation optic flow.

This thesis investigates how team working principles can be applied to Auditory Displays (AD). During this work it was established that there the level of collaboration and team work within the AD community was low and that this community would benefit from a enhanced collaborative approach. The increased use of collaborative techniques will benefit the AD community by increasing quality, knowledge transfer, synergy, and enhancing innovation. The reader is introduced to a novel approach to collaborative AD entitled Multi-listener Auditory Displays (MLAD). This work focused upon two areas of MLAD distributed AD teams and virtual AD teams. A distributed AD team is a team of participants who work upon a common task at different times and locations. The distributed approach was found to work effectively when designing ADs that work upon large scale data sets such as that found in big data. A virtual AD team is a group of participants who work upon a common task simultaneously and in separate locations. A virtual AD team is assisted by computer technology such as video conferencing and email. The virtual auditory display team was found to work well by enabling a team to work more effectively together who were geographically spread. Two pilot studies are included; SonicSETI is an example of a distributed AD team, where a remote group of listeners have background white noise playing, and use passive listening to detect anomalous candidate signals; and a geographically diverse virtual AD team that collaborates through electronic technology on an auditory display which sonifies a database of red wine measurements. A workshop was organised at a conference which focused upon ensemble auditory displays with a group of participants who were co- located.

Auditory graphs and active point estimation provide an inexpensive, accessible alternative for low vision and blind K-12 students using number lines and coordinate graphs. In the first phase of this research program, a series of four psychophysics studies demonstrated an interactive auditory number line that enables blind, low vision, and sighted people to find small targets with a laptop, headphones, and a mouse or a keyboard. The Fitts' Law studies showed that, given appropriate auditory feedback, blind people can use a mouse. In addition, auditory feedback can generate target response patterns similar to when people use visual feedback. Phase two introduced SQUARE, a novel method for building accessible alternatives to existing education technologies. The standards-driven and teacher-directed approach generated 17 graphing standards for sixth grade mathematics, all of which emphasized point estimation. It also showed that how only few basic behavioral components are necessary for these graphing problems. The third phase evaluated active point estimation tools in terms of training, classroom situations, and a testing situation. This work shows that students can learn to graph in K-12 environments, regardless of their visual impairment. It also provides several technologies used for graphing, and methods to further develop education accessibility research.

A constantly increasing number of advanced driver assistance systems are implemented in cars.The driver is required to interpret the data reported by these systems while driving, and thereforeit is important to know how the information signals provided by such systems affect the driver.Driving relies to a large extent on vision, and most current advanced driver assistance systemsuse visual information, requiring visual attention from the driver. This conflicts with the drivingtask and leads to an increase in inattention and in cognitive and visual workload. Off-loadingvisually demanding tasks to other sensory modalities should be beneficial in terms of the driver’sability to divide attention resources. The aim of this thesis is to gain knowledge on how the driveris affected by information and interaction with driver assistance systems utilising 3D auditorydisplays. The goal is to provide knowledge of how to create robust interfaces that assist the driverwhile maintaining the necessary safety aspects while driving.The first paper investigated the importance of sound source placement within the car compartment.Since sound can be a very effective means of capturing attention, it could cause dangeroussituations if used in the wrong way. If sound is used to draw attention, one must be sure to notdraw the driver’s attention away from something more critical. The study showed that soundcan improve attention to the road when the sound source location is in front of the driver.Previous research has shown that a combination of sensory modalities increase the robustnessof perception. To avoid additional load on the visual modality, the second paper investigatedthe usefulness of vibrotactile information, in combination with auditory and/or visual signals.There was no significant increase in performance when adding vibration to a sound signal, butvibration and sound on their own led to similar performance. It can be argued, however, thatcombinations are still useful since they reduce the risk of an important signal being missed orneglected. In this investigation, all signals containing a visual component provided the fastestresponse time.The third paper investigated the usefulness of a 3D auditory display applied on a parking assistancesystem. A user study was conducted to evaluate drivers’ workload and performancewhen using different system implementations. It was found that a 3D auditory display was wellreceived as long as the sound only came from one direction at a time. Providing too much informationto the driver by using simultaneous sound sources at different locations was perceived asconfusing and annoying. In some cases, the effort required to perform a successful parking wassignificantly reduced when using a 3D auditory display system compared to a traditional parkingassistance system using only two sound sources. There were also indications that 3D soundcan reduce drivers’ mental workload and frustration with parking assistance systems.Placing sound sources in the car where they are most suitable can be difficult, especially if soundsources outside the car are required. The fourth paper investigated if it would be possible to use acrosstalk cancellation technique in combination with binaural recordings to reproduce 3D audiousing only two loudspeakers in the car compartment. It was found that when the loudspeakersare placed close to the ears of the driver, crosstalk cancellation can be effective inside the car compartment by avoiding strong reflections. Closely placed loudspeakers reduce filter creationcomplexity and improve sound quality.The fifth paper tackles the problems of within-the-head perception and front-back confusion,commonly found in binaural recordings and synthesis. It has been shown that reverberant binauralsounds are perceived more externalised than anechoic sounds, and tend to suffer less fromfront-back confusion. However, information sound signals should rarely be reverberant. Theidea was to create an image-source model with focus on early reflections and compare it withtraditional artificial head recordings. The image-source model performed better than the artificialhead recordings in terms of front-back confusion, and at the same time considerably limitedthe inherent room acoustics in the sound.In conclusion, meaningful and informative sound signals can raise driving performance if theyare placed in front of the driver. This encourages the driver to maintain focus on the road, byusing the attention capture capabilities of sound. A 3D auditory display can be beneficial forsystems such as parking assistance, since it helps lowering driver effort, mental workload andfrustration. Also, 3D auditory displays increase the drivers’ sense of safety and comfort, by augmentingthe drivers’ surrounding and thereby increasing their situational awareness. If a 3Dauditory display should be implemented in the car, a transaural system with crosstalk cancellationshowed promise for virtual sound source reproduction. However, binaural synthesis hasinherent problems that need to be addressed, specifically front-back confusion. It is hard to recommendusing crosstalk cancellation with binaural synthesis at the current state for locationcritical signals due to localisation confusion. However, using more than one modality for informationsignals makes the signals difficult to miss and could also help with the front-back confusionproblem, but further research is needed. If a transaural system with crosstalk cancellationis chosen as reproduction format, make sure to carefully select the loudspeaker placementsin order to avoid as many problems with filter generation as possible, since this will cause errorsin crosstalk suppression and affect the reproduction sound quality. Directive loudspeakersmounted in the neck rest, or ceiling, are good candidates for implementation.

Many television experiences depend on users being both willing and able to visually attend to screen-based information. Auditory displays offer an alternative method for presenting this information and could benefit all users. This thesis explores how this may be achieved through the design and evaluation of auditory displays involving varying degrees of concurrency for two television use cases: menu navigation and presenting related content alongside a television show. The first study, on the navigation of auditory menus, looked at onset asynchrony and word length in the presentation of spoken menus. The effects of these on task duration, accuracy and workload were considered. Onset asynchrony and word length both caused significant effects on task duration and accuracy, while workload was only affected by onset asynchrony. An optimum asynchrony was identified, which was the same for both long and short words, but better performance was obtained with the shorter words that no longer overlapped. The second experiment investigated how disruption, workload, and preference are affected when presenting additional content accompanying a television programme. The content took the form of sound from different spatial locations or as text on a smartphone and the programme's soundtrack was either modified or left unaltered. Leaving the soundtrack unaltered or muting it negatively impacted user experience. Removing the speech from the television programme and presenting the secondary content as sound from a smartphone was the best auditory approach. This was found to compare well with the textual presentation, resulting in less visual disruption and imposing a similar workload. Additionally, the thesis reviews the state-of-the-art in television experiences and auditory displays. The human auditory system is introduced and important factors in the concurrent presentation of speech are highlighted. Conclusions about the utility of concurrency within auditory displays for television are made and areas for further work are identified.

Designing auditory interfaces is a challenge for current human-systems developers. This is largely due to a lack of theoretical guidance for directing how best to use sounds in today's visually-rich graphical user interfaces. This dissertation provided a framework for guiding the design of audio interfaces to enhance human-systems performance. This doctoral research involved reviewing the literature on conveying temporal and spatial information using audio, using this knowledge to build three theoretical models to aid the design of auditory interfaces, and empirically validating select components of the models. The three models included an audio integration model that outlines an end-to-end process for adding sounds to interactive interfaces, a temporal audio model that provides a framework for guiding the timing for integration of these sounds to meet human performance objectives, and a spatial audio model that provides a framework for adding spatialization cues to interface sounds. Each model is coupled with a set of design guidelines theorized from the literature, thus combined, the developed models put forward a structured process for integrating sounds in interactive interfaces. The developed models were subjected to a three phase validation process that included review by Subject Matter Experts (SMEs) to assess the face validity of the developed models and two empirical studies. For the SME review, which assessed the utility of the developed models and identified opportunities for improvement, a panel of three audio experts was selected to respond to a Strengths, Weaknesses, Opportunities, and Threats (SWOT) validation questionnaire. Based on the SWOT analysis, the main strengths of the models included that they provide a systematic approach to auditory display design and that they integrate a wide variety of knowledge sources in a concise manner. The main weaknesses of the models included the lack of a structured process for amending the models with new principles, some branches were not considered parallel or completely distinct, and lack of guidance on selecting interface sounds. The main opportunity identified by the experts was the ability of the models to provide a seminal body of knowledge that can be used for building and validating auditory display designs. The main threats identified by the experts were that users may not know where to start and end with each model, the models may not provide comprehensive coverage of all uses of auditory displays, and the models may act as a restrictive influence on designers or they may be used inappropriately. Based on the SWOT analysis results, several changes were made to the models prior to the empirical studies. Two empirical evaluation studies were conducted to test the theorized design principles derived from the revised models. The first study focused on assessing the utility of audio cues to train a temporal pacing task and the second study combined both temporal (i.e., pace) and spatial audio information, with a focus on examining integration issues. In the pace study, there were four different auditory conditions used for training pace: 1) a metronome, 2) non-spatial auditory earcons, 3) a spatialized auditory earcon, and 4) no audio cues for pace training. Sixty-eight people participated in the study. A pre- post between subjects experimental design was used, with eight training trials. The measure used for assessing pace performance was the average deviation from a predetermined desired pace. The results demonstrated that a metronome was not effective in training participants to maintain a desired pace, while, spatial and non-spatial earcons were effective strategies for pace training. Moreover, an examination of post-training performance as compared to pre-training suggested some transfer of learning. Design guidelines were extracted for integrating auditory cues for pace training tasks in virtual environments. In the second empirical study, combined temporal (pacing) and spatial (location of entities within the environment) information were presented. There were three different spatialization conditions used: 1) high fidelity using subjective selection of a "best-fit" head related transfer function, 2) low fidelity using a generalized head-related transfer function, and 3) no spatialization. A pre- post between subjects experimental design was used, with eight training trials. The performance measures were average deviation from desired pace and time and accuracy to complete the task. The results of the second study demonstrated that temporal, non-spatial auditory cues were effective in influencing pace while other cues were present. On the other hand, spatialized auditory cues did not result in significantly faster task completion. Based on these results, a set of design guidelines was proposed that can be used to direct the integration of spatial and temporal auditory cues for supporting training tasks in virtual environments. Taken together, the developed models and the associated guidelines provided a theoretical foundation from which to direct user-centered design of auditory interfaces.
Ph.D.
Department of Industrial Engineering and Management Systems
Engineering and Computer Science
Industrial Engineering PhD

Thesis (S.M.)--Massachusetts Institute of Technology, Engineering Systems Division, 2008.
This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.
Includes bibliographical references (p. 101-105).
With advanced autonomy, Unmanned Aerial Vehicle (UAV) operations will likely be conducted by single operators controlling multiple UAVs. As operator attention is divided across multiple supervisory tasks, there is a need to support the operator's awareness of the state of the tasks for safe and effective task management. This research explores enhancing audio cues of UAV interfaces for this futuristic control of multiple UAVs by a single operator. This thesis specifically assesses the value of continuous and discrete audio cues as indicators of course-deviations or late-arrivals to targets for UAV missions with single and multiple UAVs. In particular, this thesis addresses two questions: (1) when compared with discrete audio, does continuous audio better aid human supervision of UAV operations, and (2) is the effectiveness of the discrete or continuous audio support dependent on operator workload? An experiment was carried out on the Multiple Autonomous Unmanned Vehicle Experiment (MAUVE) test bed with 44 military participants. Specifically, two continuous audio alerts were mapped to two human supervisory tasks within MAUVE. These continuous alerts were tested against single beep discrete alerts. The results show that the use of the continuous audio alerts enhances a single operator's performance in monitoring single and multiple, semi-autonomous vehicles. The results also emphasize the necessity to properly integrate the continuous audio with other auditory alarms and visual representations in a display, as it is possible for discrete audio alerts to be masked by continuous audio, leaving operators reliant on the visual aspects of the display.
by Hudson D. Graham.
S.M.

In the driving task, vision, hearing, and the haptic senses are all used by the driver to gather required information. Future Intelligent Transportation Systems components are likely to further increase the volume of information available to or required by the driver, particularly in the case of commercial vehicle operators. The use of alternate modalities to present in-vehicle information is a possible solution to the potential overload of the visual channel. Auditory icons have been shown to improve operator performance and decrease learning and response times, not only in industrial applications, but also as emergency braking warnings. The use of auditory icons in commercial truck cabs has the potential to increase the number of auditory displays that can be distinguished and understood by commercial vehicle operators, and this experiment sought to determine the utility of auditory icons in that situation. Nine auditory icons were evaluated by commercial vehicle operators as they drove an experimental vehicle over public roads. A comparison of the data collected in the truck-cab environment to data collected in a laboratory study on the same auditory icons revealed some differences in the perceived meaning, perceived urgency, and association with the auditory icons' intended meanings between the two conditions. The presence of these differences indicates that driver evaluations of auditory icons can be affected by the environment, and testing should therefore be conducted in a situation that approximates the end-user environment as closely as possible. A comparison of the drivers' brake response times across the three warning conditions (no warning, auditory icon, and soft braking) was also conducted on a closed, secure handling course. Dependent measures included overall brake reaction time and its components, steering response time, time to initial driver action, and categorical measures of driver responses (steering, swerving, braking, and stopping). The results indicated numerically shorter mean response times (on the order of 0.5 seconds for Total Brake Response Time) for the two conditions with warnings, but the differences were not statistically significant. The most likely reason for this lack of significance is the extreme between-subject variability in response times in the no warning condition. An analysis of the response time variance across the three conditions did indicate significantly less variability in operator responses in the two warning conditions. Two of the five dependent measures (Brake Pedal Contact Time and Total Brake Response Time) exhibited significantly reduced variance in the auditory icon warning condition compared to the no warning condition. The soft braking warning condition exhibited significantly reduced variance for four of the dependent measures (Accelerator Reaction Time, Brake Pedal Contact Time, Total Brake Response Time, and First Reaction Time). These results indicate that a soft braking stimulus like that used in this study could potentially prove to be a more effective emergency braking warning than simple auditory warnings alone.
Master of Science

This thesis analyses if the change of auditory feedback can improve the effectiveness of performance in the interaction with a non-visual system, or with a system used by individuals with visual impairment. Two prototypes were developed, one with binaural audio and the other with stereo audio. The interaction was evaluated in an experiment where 22 participants, divided into two groups, performed a number of interaction tasks. A post-interview were conducted together with the experiment. The result of the experiment displayed that there were no great difference between binaural audio and stereo regarding the speed and accuracy of the interaction. The post-interviews displayed interesting differences in the way participants visualized the virtual environment that affected the interaction. This opened up interesting questions for future studies.

The Web has opened up many possibilities for disabled people to interact with society, but there is unfortunately a lack of parity between the user interface presented to different users.

This dissertation presents a proof of concept on designing a spatial layout presentation for blind users using a screen reader. This is done in three steps by first conducting a survey to determine current practices of web developers, then implementing an instant spatial feedback and comparison function that present the spatial layout, and ends with an evaluation of the spatial layout presentation by the way of user testing

The survey yielded a set of guidelines for the realistic development of web technologies for disabled persons based on the participants answered. From the implementation a concept for spatial feedback functions that are portable and expandable is presented. The evaluation shows that the created spatial presentation method passes both objectively and subjectively.

A human factors experiment was conducted to investigate the potential benefits of using directional auditory cues in intelligent transportation system technologies in commercial vehicles. Twelve licensed commercial vehicle operators drove a commercial truck-driving simulator and were prompted to select highway numbers on a side-task display. Prompts were presented visually or aurally. Auditory prompts were presented either diotically (both ears simultaneously) or directionally (to either the left or right ear). The search task varied in map density and timing of the prompts in relation to speed limit changes. All experimental conditions were compared to a control condition containing no secondary task. Both driving performance (lane deviation, steering wheel angle, road heading angle error, accidents, and adherence to the speed limit) and secondary task performance (accuracy and response time) measures were collected. Results showed that drivers were able to respond more quickly and accurately to the search task when directional auditory cues were used. Results also showed that driving performance degrades when display density increases and that the use of directional auditory prompts lessens this deterioration of performance for high-density conditions.
Master of Science

Thesis (Ph.D)--Psychology, Georgia Institute of Technology, 2010.
Committee Chair: Walker, Bruce; Committee Member: Bonebright, Terri; Committee Member: Catrambone, Richard; Committee Member: Corso, Gregory; Committee Member: Rogers, Wendy. Part of the SMARTech Electronic Thesis and Dissertation Collection.

In a large number of electronic devices, users interact with the system by navigating through various menus. Auditory menus can complement or even replace visual menus, so research on auditory menus has recently increased with mobile devices as well as desktop computers. Despite the potential importance of auditory displays on touch screen devices, little research has been attempted to enhance the effectiveness of auditory menus for those devices. In the present study, I investigated how advanced auditory cues enhance auditory menu navigation on a touch screen smartphone, especially for new input gestures such as tapping, wheeling, and flicking methods for navigating a one-dimensional menu. Moreover, I examined if advanced auditory cues improve user experience, not only for visuals-off situations, but also for visuals-on contexts. To this end, I used a novel auditory menu enhancement called a "spindex" (i.e., speech index), in which brief audio cues inform the users of where they are in a long menu. In this study, each item in a menu was preceded by a sound based on the item's initial letter. One hundred and twenty two undergraduates navigated through an alphabetized list of 150 song titles. The study was a split-plot design with manipulated auditory cue type (text-to-speech (TTS) alone vs. TTS plus spindex), visual mode (on vs. off), and input gesture style (tapping, wheeling, and flicking). Target search time and subjective workload for the TTS + spindex were lower than those of the TTS alone in all input gesture types regardless of visual type. Also, on subjective ratings scales, participants rated the TTS + spindex condition higher than the plain TTS on being 'effective' and 'functionally helpful'. The interaction between input methods and output modes (i.e., auditory cue types) and its effects on navigation behaviors was also analyzed based on the two-stage navigation strategy model used in auditory menus. Results were discussed in analogy with visual search theory and in terms of practical applications of spindex cues.

In the world of ICT, strategies for innovation and development are increasingly focusing on applications that require spatial representation and real-time interaction with and within 3D media environments. One of the major challenges that such applications have to address is user-centricity, reflecting e.g. on developing complexity-hiding services so that people can personalize their own delivery of services. In these terms, multimodal interfaces represent a key factor for enabling an inclusive use of the new technology by everyone. In order to achieve this, multimodal realistic models that describe our environment are needed, and in particular models that accurately describe the acoustics of the environment and communication through the auditory modality. Examples of currently active research directions and application areas include 3DTV and future internet, 3D visual-sound scene coding, transmission and reconstruction and teleconferencing systems, to name but a few. The concurrent presence of multimodal senses and activities make multimodal virtual environments potentially flexible and adaptive, allowing users to switch between modalities as needed during the continuously changing conditions of use situation. Augmentation through additional modalities and sensory substitution techniques are compelling ingredients for presenting information non-visually, when the visual bandwidth is overloaded, when data are visually occluded, or when the visual channel is not available to the user (e.g., for visually impaired people). Multimodal systems for the representation of spatial information will largely benefit from the implementation of audio engines that have extensive knowledge of spatial hearing and virtual acoustics. Models for spatial audio can provide accurate dynamic information about the relation between the sound source and the surrounding environment, including the listener and his/her body which acts as an additional filter. Indeed, this information cannot be substituted by any other modality (i.e., visual or tactile). Nevertheless, today's spatial representation of audio within sonification tends to be simplistic and with poor interaction capabilities, being multimedia systems currently focused on graphics processing mostly, and integrated with simple stereo or multi-channel surround-sound. On a much different level lie binaural rendering approaches based on headphone reproduction, taking into account that possible disadvantages (e.g. invasiveness, non-flat frequency responses) are counterbalanced by a number of desirable features. Indeed, these systems might control and/or eliminate reverberation and other acoustic effects of the real listening space, reduce background noise, and provide adaptable and portable audio displays, which are all relevant aspects especially in enhanced contexts. Most of the binaural sound rendering techniques currently exploited in research rely on the use of Head-Related Transfer Functions (HRTFs), i.e. peculiar filters that capture the acoustic effects of the human head and ears. HRTFs allow loyal simulation of the audio signal that arrives at the entrance of the ear canal as a function of the sound source's spatial position. HRTF filters are usually presented under the form of acoustic signals acquired on dummy heads built according to mean anthropometric measurements. Nevertheless, anthropometric features of the human body have a key role in HRTF shaping: several studies have attested how listening to non-individual binaural sounds results in evident localization errors. On the other hand, individual HRTF measurements on a significant number of subjects result both time- and resource-expensive. Several techniques for synthetic HRTF design have been proposed during the last two decades and the most promising one relies on structural HRTF models. In this revolutionary approach, the most important effects involved in spatial sound perception (acoustic delays and shadowing due to head diffraction, reflections on pinna contours and shoulders, resonances inside the ear cavities) are isolated and modeled separately with a corresponding filtering element. HRTF selection and modeling procedures can be determined by physical interpretation: parameters of each rendering blocks or selection criteria can be estimated from real and simulated data and related to anthropometric geometries. Effective personal auditory displays represent an innovative breakthrough for a plethora of applications and structural approach can also allow for effective scalability depending on the available computational resources or bandwidth. Scenes with multiple highly realistic audiovisual objects are easily managed exploiting parallelism of increasingly ubiquitous GPUs (Graphics Processing Units). Building individual headphone equalization with perceptually robust inverse filtering techniques represents a fundamental step towards the creation of personal virtual auditory displays (VADs). To this regard, several examples might benefit from these considerations: multi-channel downmix over headphones, personal cinema, spatial audio rendering in mobile devices, computer-game engines and individual binaural audio standards for movie and music production. This thesis presents a family of approaches that overcome the current limitations of headphone-based 3D audio systems, aiming at building personal auditory displays through structural binaural audio models for an immersive sound reproduction. The resulting models allow for an interesting form of content adaptation and personalization, since they include parameters related to the user's anthropometry in addition to those related to the sound sources and the environment. The covered research directions converge to a novel framework for synthetic HRTF design and customization that combines the structural modeling paradigm with other HRTF selection techniques (inspired by non-individualized HRTF selection procedures) and represents the main novel contribution of this thesis: the Mixed Structural Modeling (MSM) approach considers the global HRTF as a combination of structural components, which can be chosen to be either synthetic or recorded components. In both cases, customization is based on individual anthropometric data, which are used to either fit the model parameters or to select a measured/simulated component within a set of available responses. The definition and experimental validation of the MSM approach addresses several pivotal issues towards the acquisition and delivery of binaural sound scenes and designing guidelines for personalized 3D audio virtual environments holding the potential of novel forms of customized communication and interaction with sound and music content. The thesis also presents a multimodal interactive system which is used to conduct subjective test on multi-sensory integration in virtual environments. Four experimental scenarios are proposed in order to test the capabilities of auditory feedback jointly to tactile or visual modalities. 3D audio feedback related to user’s movements during simple target following tasks is tested as an applicative example of audio-visual rehabilitation system. Perception of direction of footstep sounds interactively generated during walking and provided through headphones highlights how spatial information can clarify the semantic congruence between movement and multimodal feedback. A real time, physically informed audio-tactile interactive system encodes spatial information in the context of virtual map presentation with particular attention to orientation and mobility (O&M) learning processes addressed to visually impaired people. Finally, an experiment analyzes the haptic estimation of size of a virtual 3D object (a stair-step) whereas the exploration is accompanied by a real-time generated auditory feedback whose parameters vary as a function of the height of the interaction point. The collected data from these experiments suggest that well-designed multimodal feedback, exploiting 3D audio models, can definitely be used to improve performance in virtual reality and learning processes in orientation and complex motor tasks, thanks to the high level of attention, engagement, and presence provided to the user. The research framework, based on the MSM approach, serves as an important evaluation tool with the aim of progressively determining the relevant spatial attributes of sound for each application domain. In this perspective, such studies represent a novelty in the current literature on virtual and augmented reality, especially concerning the use of sonification techniques in several aspects of spatial cognition and internal multisensory representation of the body. This thesis is organized as follows. An overview of spatial hearing and binaural technology through headphones is given in Chapter 1. Chapter 2 is devoted to the Mixed Structural Modeling formalism and philosophy. In Chapter 3, topics in structural modeling for each body component are studied, previous research and two new models, i.e. near-field distance dependency and external-ear spectral cue, are presented. Chapter 4 deals with a complete case study of the mixed structural modeling approach and provides insights about the main innovative aspects of such modus operandi. Chapter 5 gives an overview of number of a number of proposed tools for the analysis and synthesis of HRTFs. System architectural guidelines and constraints are discussed in terms of real-time issues, mobility requirements and customized audio delivery. In Chapter 6, two case studies investigate the behavioral importance of spatial attribute of sound and how continuous interaction with virtual environments can benefit from using spatial audio algorithms. Chapter 7 describes a set of experiments aimed at assessing the contribution of binaural audio through headphones in learning processes of spatial cognitive maps and exploration of virtual objects. Finally, conclusions are drawn and new research horizons for further work are exposed in Chapter 8.
Il settore dell'Information and Communications Technology (ICT) sta investendo in strategie di innovazione e sviluppo sempre più rivolte ad applicazioni capaci di interazione complesse grazie alla rappresentazione spaziale in ambienti virtuali multimodali capaci di rispettare i vincoli di tempo reale. Una delle principali sfide da affrontare riguarda la centralità dell'utente, che si riflette, ad esempio, sullo sviluppo di servizi la cui complessità tecnologica viene nascosta al destinatario, e la cui offerta di servizi sia personalizzabile dall’utente e per l’utente. Per queste ragioni , le interfacce multimodali rappresentano un elemento chiave per consentire un uso diffuso di queste nuove tecnologie. Per raggiungere questo obiettivo è necessario ottenere dei modelli multimodali realistici che siano capaci di descrivere l’ambiente circostante, e in particolare modelli che sappiano rappresentare accuratamente l'acustica dell'ambiente e la trasmissione di informazione attraverso la modalità uditiva. Alcuni esempi di aree applicative e direzioni di ricerca attive nella comunità scientifica internazionale includono 3DTV e internet del futuro , codifica, trasmissione e ricostruzione della scena 3D video e audio e sistemi di teleconferenza , per citarne solo alcuni. La presenza concomitante di più modalità sensoriali e la loro integrazione rendono gli ambienti virtuali multimodali potenzialmente flessibili e adattabili, permettendo agli utenti di passare dall’una all’altra modalità in base alle necessità dettata dalle mutevoli condizioni di utilizzo di tali sistemi. Modalità sensoriali aumentata attraverso altri sensi e tecniche di sostituzione sensoriale sono elementi essenziali per la veicolazione dell’informazioni non visivamente, quando, ad esempio, il canale visivo è sovraccaricato, quando i dati sono visivamente ostruiti, o quando il canale visivo non è disponibile per l'utente (ad esempio, per le persone non vedenti). I sistemi multimodali per la rappresentazione delle informazioni spaziali beneficano sicuramente della realizzazione di motori audio che possiedano una conoscenza approfondita degli aspetti legati alla percezione spaziale e all’acustica virtuale. I modelli per il rendering di audio spazializzato sono in grado di fornire accurate informazioni dinamiche sulla relazione tra la sorgente sonora e l'ambiente circostante , compresa l'interazione del corpo dell’ascoltatore che agisce da ulteriore filtraggio acustico. Queste informazioni non possono essere sostituite da altre modalità (ad esempio quella visiva o tattile). Tuttavia , la rappresentazione spaziale del suono nei feedback acustici tende ad essere, al giorno d’oggi, semplicistica e con scarse capacità di interazione, questo perchè i sistemi multimediali attualmente si focalizzano per lo più sull’elaborazione grafica, e si accontentano di semplici tecnologie stereofoniche o surround multicanale per il rendering del suono. Il rendering binaurale riprodotto in cuffia rappresenta un approccio avveniristico, tenendo conto che i possibili svantaggi (es. invasività , risposte in frequenza non piane) possono essere man mano gestiti e controbilanciati da una serie di desiderabili caratteristiche. Questi sistemi sono caratterizzati dalla possibilità di controllare e/o eliminare il riverbero e altri effetti acustici dello spazio di ascolto circostante, di ridurre il rumore di fondo e fornire dei display audio adattabili e portatili, tutti aspetti rilevanti soprattutto in contesti di innovazione. La maggior parte delle tecniche di rendering binaurale impiegate oggigiorno in ricerca si basano sull'uso di Head Related Transfer Functions (HRTFs), vale a dire di filtri particolari che catturano gli effetti acustici di testa, busto e orecchie dell’ascoltatore. Le HRTF permettono una simulazione fedele del segnale audio che si presenta all'ingresso del canale uditivo in funzione della posizione spaziale della sorgente sonora. I filtri basati su HRTF sono generalmente presentati sotto forma di segnali acustici misurati a partire da una testa di manichino costruito secondo misurazioni antropometriche medie. Tuttavia, le caratteristiche antropometriche individuali hanno un ruolo fondamentale nel determinare le HRTF: diversi studi hanno riscontrato come l’ascolto di audio binaurale non individuale produce errori di localizzazione evidenti . D'altra parte , le misurazioni individuali di HRTF su un numero significativo di soggetti richiedono un impiego di risorse e tempo non trascurabili. Sono state proposte negli ultimi due decenni diverse tecniche per il design di HRTF sintetiche e tra le più promettente vi è quella che utilizza i modelli strutturali di HRTF. In questo approccio rivoluzionario, gli effetti più importanti coinvolti nella percezione spaziale del suono (i ritardi acustici e le ombre acustiche ad opera della diffrazione attorno alla testa, le riflessioni sui contorni dell’orecchio esterno e sulle spalle, le risonanze all'interno delle cavità dell’orecchio) sono isolati e modellati separatamente nell’elemento filtrante corrispondente. La selezione di HRTF non individuali e queste procedure di modellazione possono essere entrambe analizzate con una interpretazione fisica: i parametri di ogni blocco di rendering o i criteri di selezione possono venir stimati dalla relazione tra dati reali e simulati e antropometria dell’ascoltatore. La realizzazione di efficaci display uditivi personali rappresenta un notevole passo in avanti per numerose applicazioni; l’approccio strutturale consente una intrinseca scalabilità a seconda delle risorse computazionali o della larghezza di banda disponibili. Scene altamente realistiche con più oggetti audiovisivi riescono ad essere gestite sfruttando il parallelismo della Graphics Processing Unit (GPU) sempre più onnipresenti. Ottenere un equalizzazione individuale delle cuffie con tecniche di filtraggio inverso che siano percettivamente robuste costituisce un passo fondamentale verso la creazione di display uditivi virtuali personali. A titolo d’esempio, vengono di seguito riportate alcune aree applicative che possono trarre beneficio da queste considerazioni: riproduzione multi canale in cuffia, rendering spaziale del suono in dispositivi mobile, motori di rendering per computer-game e standard audio binaurali individuali per film e produzione musicale. Questa tesi presenta una famiglia di approcci in grado di superare gli attuali limiti dei sistemi di audio 3D in cuffia, con l’obiettivo di realizzare display uditivi personali attraverso modelli strutturali per l’audio binaurale volti ad una riproduzione immersiva del suono. I modelli che ne derivano permettono adattamento e personalizzazione di contenuti, grazie alla gestione dei parametri relativi all’antropometria dell'utente oltre a quelli relativi alle sorgenti sonore nell'ambiente . Le direzioni di ricerca intraprese convergono verso una metodologia per la progettazione e personalizzazione di HRTF sintetiche che unisce il paradigma di modellazione strutturale con altre tecniche di selezione per HRTF (ispirate a procedure di selezione non-individuali di HRTF) e rappresenta il principale contributo di questa tesi: l’ approccio a modellazione strutturale mista( MSM ) che considera la HRTF globale come una combinazione di elementi strutturali, che possono essere scelti tra componenti sia sintetiche che registrate. In entrambi i casi, la personalizzazione si basa su dati antropometrici individuali, utilizzati per adattare sia i parametri del modello sia per selezionare un componente simulato o misurato, tra un insieme di risposte all’impulso disponibili. La definizione e la validazione sperimentale dell'approccio a MSM affronta alcune questioni cruciali riguarda l'acquisizione e il rendering di scene acustiche binaurali, definendo alcune linee guida di progettazione per ambienti virtuali personali che utilizzano l’audio 3D e che possiedono nuove forme di comunicazione su misura e di interazione con contenuti sonori e musicali. In questa tesi viene anche presentato un sistema interattivo multimodale utilizzato per condurre test soggettivi sull’integrazione multisensoriale in ambienti virtuali. Vengono proposti quattro scenari sperimentali al fine di testare le funzionalità di un feedback sonoro integrato a modalità tattili o visive. (i) Un feedback con audio 3D legato ai movimenti dell'utente durante una semplice attività di inseguimento di un bersaglio viene presentato come un esempio applicativo di sistema riabilitativo audiovisivo. (ii) La percezione della direzione sonora dei passi interattivamente generati in cuffia durante la camminata evidenzia come l'informazione spaziale sia in grado di mettere in luce la congruenza semantica tra movimento e feedback multimodale. (iii) Un sistema audio tattile interattivo e real-time sintetizza l'informazione spaziale di mappe virtuali per l’educazione all’orientamento e alla mobilità (O&M) rivolta a persone non vedenti. (iv) Un ultimo esperimento analizza la stima tattile delle dimensioni di un oggetto virtuale 3D (un gradino), mentre l'esplorazione è accompagnata da un feedback sonoro generato in tempo reale i cui parametri variano in funzione dell’altezza del punto di interazione aptico. I dati raccolti da questi esperimenti suggeriscono che feedback multimodali che sfruttano correttamente modelli di audio 3D, possono essere utilizzati per migliorare la navigazione nella realtà virtuale, l’orientamento e l’apprendimento di azioni motorie complesse, grazie all'alto livello di attenzione, impegno e immersività fornito all'utente. La metodologia di ricerca, basata sull'approccio a MSM, rappresenta un importante strumento di valutazione per determinare progressivamente i principali attributi spaziali del suono in relazione a ciascun dominio applicativo. In questa prospettiva, tali studi rappresentano una novità nella letteratura scientifica corrente che ha come principale argomento di indagine la realtà virtuale e aumentata, soprattutto per quanto riguarda l'uso di tecniche di sonicazione legate alla cognizione spaziale e alla rappresentazione multisensoriale interna del corpo . Questa tesi è organizzata come segue. Un’introduzione e una panoramica sulla percezione spaziale del suono e sulle tecnologie binaurali in cuffia sono fornite nel Capitolo 1. Il Capitolo 2 è dedicato al formalismo sulla modellazione strutturale mista e sua corrispondente filosofia di ricerca. Nel Capitolo 3 vengono presentati i modelli strutturali relativi ad ogni parte del corpo, risultanti da precedenti ricerche. Due nuove proposte di modello di testa e orecchio approfondiscono rispettivamente la dipendenza dalla distanza nel near-field e le informazioni spettrali fornite dall’orecchio esterno per la localizzazione verticale del suono. Il Capitolo 4 si occupa di un caso di studio completo riguardante l'approccio a modellazione strutturale mista, fornendo degli approfondimenti riguardanti i principali aspetti innovativi di tale modus operandi. Il Capitolo 5 fornisce una panoramica di strumenti sviluppati per l'analisi e la sintesi di HRTF. Inoltre linee guida per il design di ambienti di realtà virtuale vengono discussi in termini di problematiche riguardanti vincoli di tempo reali, requisiti per la mobilità e personalizzazione del segnale audio. Nel Capitolo 6, attraverso due casi di studio viene approfondita l'importanza dell'attributo spaziale del suono nel comportamento dell’ascoltatore e come la continua interazione in ambienti virtuali possa utilizzare con successo algoritmi per l’audio spaziale. Il Capitolo 7 descrive una serie di esperimenti volti a valutare il contributo dell’audio binaurale in cuffia in processi di apprendimento di mappe cognitive spaziali e nell'esplorazione di oggetti virtuali. Infine, il Capitolo 8 apre a nuovi orizzonti per futuri lavori di ricerca.