Academic literature on the topic 'Audio immersif'

Penelitian ini bertujuan untuk menggabungkan konsep pembelajaran imersif pada pendidikan anak usia dini (PAUD) dengan pengenalan tempat ibadah umat Buddha di Pontianak menggunakan Virtual Reality. Metodologi pengembangan penelitian ini antara lain membuat Virtual Reality tempat ibadah agama Buddha di Pontianak yang interaktif dan detail serta mengevaluasi efektivitasnya dalam meningkatkan pemahaman agama Buddha pada anak PAUD. Virtual Reality dikembangkan menggunakan teknologi multimedia termasuk gambar, audio, dan video 360 derajat untuk memberikan pengalaman mendalam bagi anak-anak. Temuan menunjukkan bahwa pemanfaatan Virtual Reality dalam konteks pembelajaran imersif di PAUD efektif meningkatkan pemahaman anak terhadap tempat ibadah agama Buddha di Pontianak. Anak-anak menunjukkan minat yang besar dan partisipasi aktif dalam menggunakan Virtual Reality , dan guru serta orang tua mengamati adanya peningkatan dalam pemahaman anak-anak tentang konsep-konsep Buddhis. Kajian ini menyoroti pentingnya memasukkan teknologi imersif ke dalam kurikulum PAUD untuk menanamkan nilai-nilai agama dan keberagaman budaya pada anak sejak dini. Implikasi praktis dari penelitian ini adalah memberikan alternatif pembelajaran yang menarik dan menyenangkan bagi anak PAUD sekaligus mengenalkan mereka pada keberagaman agama yang ada di masyarakatnya.

Penelitian ini mengeksplorasi pemanfaatan Ruang Kelas Virtual Reality (VR) sebagai alternatif untuk konferensi video dalam pembelajaran online, dengan tujuan mengatasi kendala bandwidth. Metode penelitian yang digunakan adalah analisis komparatif antara sistem interaksi avatar VR dan sistem video konferensi tradisional. Penelitian berfokus pada pengoptimalan penggunaan bandwidth dalam VR melalui interaksi berbasis avatar. Pengumpulan data dilakukan melalui pengamatan perilaku interaksi pengguna dan pengukuran penggunaan bandwidth dalam komunikasi antar klien. Data yang terkumpul dianalisis secara kuantitatif untuk memahami perbedaan perilaku pengguna dan implikasinya terhadap pengalaman interaktif serta strategi yang diperlukan untuk memastikan komunikasi yang efisien dalam lingkungan pembelajaran daring. Hasil pengukuran menunjukkan rata-rata latensi system konferensi video sebesar 184 ms (dengan tambahan proses encoding decoding yang mengakibatkan delay hingga 1 detik) dengan rata-rata latensi VR classroom sebesar 133 ms. Sistem avatar VR menunjukkan latensi yang lebih rendah serta memberikan pengalaman belajar yang responsif dan immersif. Selain itu, sistem avatar VR menunjukkan ketangguhan dalam menghadapi keterbatasan jaringan, menjaga kualitas interaksi yang konsisten. Sistem dapat mencapai framerate stabil pada 60 fps dan menghindari desinkronisasi antara audio dan video pada tingkat limitasi bandwith 100kbps di mana system konferensi video gagal mempertahankan feed video dan audio. Namun, perlu dicatat bahwa implementasi sistem avatar VR memerlukan pengaturan dan persyaratan perangkat keras yang lebih kompleks dibandingkan dengan video konferensi.

There has been an increase in the use of immersive or spatialized audio formats for radio and podcast journalism. Immersion is used to put audiences at the heart of a story, enable richer experiences and encourage empathy with others, but it can disrupt the ‘grammar’ of broadcast formats and the codes that structure the relationship between audience, journalist and story. Immersive journalism research has not tackled the impact on audio-only storytelling, and the lack of research by and for audio journalists means programme-makers have until now lacked a conceptual framework and terminology to describe how space is constructed in immersive audio, the creative and editorial choices available and their effects. This article, based on analysis of immersive output and interviews with those who produce it, critically examines the differences between mono/stereo space and immersive audio space and argues they are not only a matter of aesthetics or comfort, but communicate differential authority over the story and merit further attention when journalists are trained in immersive audio.

We introduce and describe a new conceptual framework for the design and analysis of audio for immersive first-person shooter games, and discuss its potential implications for the development of the audio component of game engines. The framework was created in order to illustrate and acknowledge the direct role of in-game audio in shaping player-player interactions and in creating a sense of immersion in the game world. Furthermore, it is argued that the relationship between player and sound is best conceptualized theoretically as an acoustic ecology. Current game engines are capable of game world spatiality through acoustic shading, but the ideas presented here provide a framework to explore other immersive possibilities for game audio through real-time synthesis.

Augmented Reality Audio Games (ARAG) enrich the physical world with virtual sounds to express their content and mechanics. Existing ARAG implementations have focused on exploring the surroundings and navigating to virtual sound sources as the main mode of interaction. This paper suggests that gestural activity with a handheld device can realize complex modes of sonic interaction in the augmented environment, resulting in an enhanced immersive game experience. The ARAG “Audio Legends” was designed and tested to evaluate the usability and immersion of a system featuring an exploration phase based on auditory navigation, as well as an action phase, in which players aim at virtual sonic targets and wave the device to hit them or hold the device to block them. The results of the experiment provide evidence that players are easily accustomed to auditory navigation and that gestural sonic interaction is perceived as difficult, yet this does not affect negatively the system’s usability and players’ immersion. Findings also include indications that elements, such as sound design, the synchronization of sound and gesture, the fidelity of audio augmentation, and environmental conditions, also affect significantly the game experience, whereas background factors, such as age, sex, and game or music experience, do not have any critical impact.

Applying sonic interaction to game design has gained momentum in recent year. This paper proposes a novel approach to design an experimental audio game Quartet that is immersive and highly interactive in the mixed reality environment. The chief aims of designing this game are to explore the music improvisation techniques by using the software Touchdesigner and to create a new spectatorship of sonic interaction in immersive experiences. To achieve the goals, several methods were presented: interactive storytelling, audio visualization, ceding control, sonic interaction, etc. The research has also recorded valuable data about players experiences of the game including their overall level of immersion, perception of the story, and overall feeling of the sound design. The results suggested that an interactive sonic design can boost players engagement and convey emotions and information more efficiently and accurately. For further research, this audio based experiment can be contributed to the field of inclusive design especially for visually impaired people and children with multiple disabilities.

For people with disabilities being part of social life is essential for well-being. To accomplish this there are two possibilities: through interaction in person and through access to culture. One way to enjoy both possibilities is the cinema. For people with sight loss (people that are blind or visually impaired), there exists audio description which allows enjoying a film in a cinema. Project FASEA examined if this audience is as much immersed and could enjoy it as much as sighted people while watching a film with audio description, received through an app called ‘Greta’. Twenty-five people with blindness or visual impairment took part and were matched with sighted people in age, sex, education, and nationality. This was conducted in a commercial cinema under natural conditions with a comedy. We assessed immersion through the Narrative Engagement Scale (NES), the Modes of Reception Inventory (MOR), and the Audience Response Scale (ARS). Enjoyment was evaluated with the Emotions in Media Scale (EMS). People with sight loss showed lower scores in narrative understanding in NES and less thought-provoking and suspense in ARS. Also, they experienced fewer positive emotions. In all other scales, including all MOR scales, there were no differences between them and sighted people. The results confirm that audio description, provided by an application, is a useful tool for people with sight loss to immerse themselves and enjoy films as much as sighted people do and hence be part of social life.

The process of listening to an audiobook is usually a rather passive act that does not require an active interaction. If spatial interaction is incorporated into a storytelling scenario, can open. Possibilities of a novel experience which allows an active participation might affect the user-experience. The aim of this paper is to create a portable prototype system based on an embedded hardware platform, allowing listeners to get immersed in an interactive audio storytelling experience enhanced by dynamic binaural audio rendering. For the evaluation of the experience, a short story based on the horror narrative of Stephen King’s Strawberry Springs is adapted and designed in virtual environments. A comparison among three different listening experiences, namely, (i) monophonic (traditional audio story), (ii) static binaural rendering (state-of-the-art audio story), and (iii) our prototype, is conducted. We discuss the quality of the experience based on usability testing, physiological data, emotional assessments, and questionnaires for immersion and spatial presence. Results identify a clear trend for an increase in immersion with our prototype compared to traditional audiobooks, showing also an emphasis on story-specific emotions, i.e., terror and fear.

Virtual reality and 360° video are some of the latest technological developments within the media and communications industry. These technologies, which are designed to facilitate viewer immersion, are currently being used to create fictional and non-fictional content, thus giving rise to a new audio-visual narrative. On the basis of these premises, this research article analyses how immersive narratives are applied to the social documentary genre in its social dimension. To this end, qualitative content analysis was performed on a sample of 49 immersive documentaries published on the WITHIN platform. This analysis, which was completed with quantitative data, allowed us to confirm that these technologies have enabled the development of immersive narratives, which has given birth to a new type of documentary – the ‘immersive social documentary’.

L’audiodescripció (AD) ja no és considerada un servei exclusiu d’accessibilitat, dedicat a una petita fracció d’individus, sinó que s’entén com un servei inclusiu, dirigit no només a les persones amb discapacitats sensorials, sinó també a altres audiències, com els ancians o els nens. És probable que la demanda d’aquest servei creixi en un futur pròxim. Encara que en el mercat s’observa un interès en augmentar cada cop més la oferta de productes audiovisuals accessibles, no s’hauria de deixar de banda la qualitat de l’experiència de l’usuari. Emmarcada dins el model inclusiu d’accessibilitat, aquesta tesi doctoral té com a objectiu explorar l’impacte de l’AD en la resposta emocional del seu públic objectiu – usuaris cecs i amb baixa visió (CiBV). Per a això, es van realitzar dos estudis de recerca, un centrat en l’estil de AD i l’altre en el el tipus de veu d’AD. La justificació darrere de la selecció d’aquests paràmetres es va relacionar amb els resultats de la investigació anterior. Es van establir dos objectius principals d’investigació. El primer, en relació amb l’estil d’AD, es va centrar en l’estudi de la recepció de dos estils d’AD – estàndard i creatiu – per als usuaris CiBV. El segon, vinculat al tipus de veu d’AD, va implicar l’estudi de la recepció de dos tipus de veu d’AD – humana i sintètica – per a usuaris CiBV per a dos gèneres: ficció i documental. L’aspecte innovador d’aquesta tesi doctoral és l’enfocament metodològic adoptat. La tesi segueix una metodologia centrada en l’usuari, però no es limita a informar sobre les opinions dels usuaris, sinó que pretén mesurar la seva resposta emocional als estímuls presentats. L’eina d’investigació que s’utilitza per mesurar l’experiència de l’usuari és el qüestionari “Independent Television Commission Sense of Presence Inventory”, un dels cinc qüestionaris canònics utilitzats per mesurar la presència. Dos elements addicionals – interès i confusió – s’afegeixen perquè els usuaris expressin la seva resposta emocional als estímuls mostrats. Els resultats indiquen que no només els guions d’AD, creatius o estàndard, sinó també els tipus de veu d’AD tenen un efecte directe en la recepció de la pel·lícula i, per tant, en l’experiència dels usuaris. Quan es tracta de l’estil d’AD, els resultats mostren que l’AD creativa, en comparació amb l’AD estàndard, va produir nivells més alts de presència en tots els participants. En general, l’estil creatiu semblava ser més natural, especialment per als participants amb pèrdua de visió recent. No obstant això, va resultar ser més confús que l’estil estàndard, possiblement perquè els participants no estaven acostumats a la terminologia cinematogràfica inclosa en el guió o no esperaven escoltar vocabulari col·loquial. Tot i això, quan es va preguntar explícitament quin estil d’AD – creatiu o estàndard – prefereien, la majoria dels participants va optar per l’AD creativa. Quan es tracta del tipus de veu d’AD, l’AD amb veu humana, en comparació amb l’AD amb veu sintètica (TTS AD1), va provocar nivells significativament més alts de presència per la ficció. Els nivells de presència per al documental van ser similars, sense diferències estadísticament significatives, en relació amb el tipus de veu d’AD. La ficció amb AD narrada per una veu humana va ser avaluada pels participants com a més interessant i menys confusa en comparació amb la ficció narrada per una veu sintètica. En el cas del documental, els nivells d’interès i confusió eren comparables per a les dues veus. Quan se’ls va preguntar quina veu preferien per a un gènere donat, els usuaris van mostrar preferència per l’AD narrada per una persona per a la ficció i cap preferència entre l’AD amb veu humana i AD amb veu sintètica per al documental. En general, les troballes suggereixen que quan es narra adequadament, tant en termes d’estil com de veu, l’AD pot facilitar que les audiències CyBV tinguin una experiència audiovisual més atractiva. S’espera que aquesta tesi doctoral sigui un primer un pas útil per entendre els vincles entre el producte d’AD i el context de l’usuari, i per a poder dur a terme investigacions més profundes en el futur.
La audiodescripción (AD) ya no está considerada como un servicio exclusivo de accesibilidad, dedicado a una pequeña fracción de individuos, sino como un servicio inclusivo, dirigido no sólo a personas con discapacidades sensoriales, sino también a otras audiencias, como los ancianos o los niños. Es probable que la demanda de este servicio crezca en un futuro cercano. Aunque en el mercado se observa un interés en aumentar cada vez más la oferta de productos audiovisuales accesibles, no se debería dejar de lado la calidad de la experiencia del usuario. Enmarcada dentro del modelo inclusivo de accesibilidad, esta tesis doctoral tiene como objetivo explorar el impacto de la AD en la respuesta emocional de su público objetivo – usuarios ciegos y con baja visión (CyBV). Para ello, se realizaron dos estudios de investigación, uno centrado en el estilo de AD y el otro en la entrega vocal de AD. La justificación detrás de la selección de estos parámetros se relacionó con los resultados de investigación anterior. Se establecieron dos objetivos principales de investigación. La primera, en relación con el estilo de AD, se centró en estudiar la recepción de dos estilos de AD – estándar y creativo – por los usuarios CyBV. La segunda, vinculada a la entrega vocal de AD, implicó el estudio de recepción de dos tipos de voces de AD – humana y sintética – por usuarios CyBV para dos géneros: ficción y documental. El aspecto novedoso de esta tesis doctoral es el enfoque metodológico adoptado. La tesis sigue una metodología centrada en el usuario, pero no se limita a informar sobre las opiniones de los usuarios, sino que pretende medir su respuesta emocional a los estímulos presentados. La herramienta de investigación que se utiliza para medir la experiencia del usuario es el cuestionario de la Independent Television Commission Sense of Presence Inventory, que es uno de los cinco cuestionarios canónicos utilizados para medir la presencia. Dos elementos adicionales – interés y confusión – se añaden para que los usuarios expresen su respuesta emocional a los estímulos presentados. Los resultados indican que no solo los guiones de AD, creativos o estándar, sino también las voces de entrega de AD tienen un efecto directo en la recepción de la película y, por lo tanto, en la experiencia de los usuarios. Cuando se trata del estilo de la AD, los resultados muestran que la AD creativa, en comparación con la AD estándar, produjo mayores niveles de presencia para todos los participantes. En general, el estilo creativo parecía más natural, especialmente a los participantes con pérdida de visión reciente. Sin embargo, resultó ser más confuso que el estilo estándar, posiblemente porque los participantes no estaban acostumbrados a la terminología cinematográfica incluida en el guión o no esperaban escuchar vocabulario coloquial. A pesar de eso, cuando se preguntó explícitamente qué estilo de AD – creativo o estándar – preferían, la mayoría de los participantes optó por la AD creativa. En cuanto al tipo de voz de AD, la AD narrada por una persona, en comparación con la AD narrada por una voz sintética (TTS AD1), provocó niveles significativamente más altos de presencia para la ficción. Los niveles de presencia para el documental fueron similares, sin diferencias estadísticamente significativas en relación con el tipo de voz de la AD. La ficción con AD narrada por una voz humana fue evaluada por los participantes como más interesante y menos confusa en comparación con la AD para ficción narrada mediante síntesis de voz. En el caso del documental, los niveles de interés y confusión eran similares para ambas voces. Cuando se les preguntó qué voz prefierían para un género dado, los usuarios mostraron una preferencia por la AD narrada por una persona para la ficción y ninguna preferencia entre la AD con voz humana y AD con voz sintética para el documental. En general, los resultados sugieren que cuando se narra adecuadamente, tanto en términos de estilo como de voz, la AD puede aumentar las posibilidades de que las audiencias CyBV tengan una experiencia audiovisual más atractiva. Se espera que esta tesis doctoral sea un primer paso para entender los vínculos entre el producto de la AD y el contexto del usuario, y abrir el camino a investigaciones más profundas en el futuro.
Audio description (AD) is no longer seen as an exclusive accessibility service dedicated to a small fraction of individuals, but as an inclusive service, addressing not only the needs of people with sensory impairments, but also those of other audiences, including the elderly or children. The demand for this service is likely to grow in the near future. Although a drive for quantity of accessible audiovisual products can be observed on the market, the issue of the quality of user experience should not be overlooked. Framed within the inclusive model of accessibility, this PhD thesis aims to explore the impact of AD on the emotional response of its target audiences – blind and visually impaired (B/VIP) users. To this end, two research studies were carried out: one focusing on AD style and the other on AD vocal delivery. The rationale behind selecting these parameters was related to the findings of previous research conducted in the field. Two main research objectives were set. The first one, concerning AD style, centred around studying the reception of two AD styles – standard and creative – by B/VIP users. The second one, linked to AD vocal delivery, involved studying the reception of two AD voice types – human and synthetic – by B/VIP users for two genres: fiction and documentary. The novel aspect of this PhD thesis is the methodological approach adopted. It follows a user-centric methodology, not limiting itself to reporting on users’ opinions, but also aiming to measure their emotional response to a stimuli presented. The research tool that is used to gauge user experience is the Independent Television Commission Sense of Presence Inventory questionnaire, one of five canonical questionnaires used for measuring presence. Two additional items – interest and confusion – are added for users to report their emotional response to the stimuli shown. The results indicate that not only AD scripts – creative or standard – but also AD delivery voices have a direct effect on the reception of a film, and therefore on users’ experience. When it comes to AD style, the results show that creative AD, compared to standard AD, yielded higher levels of presence for all participants. Overall, the creative AD style seemed more natural, especially to participants with recent sight loss. However, it turned out to be more confusing than standard AD, possibly because the participants were not used to cinematic terminology included in the script or did not expect to hear unsavoury vocabulary. Nevertheless, when explicitly asked which AD style – creative or standard – they prefer, the majority of participants opted for creative AD. When it comes to AD vocal delivery, AD narrated by a human, compared to text-to-speech (TTS) AD, prompted significantly higher levels of presence for fiction. Presence rates for documentary were similar, with no statistically significant differences in relation to AD voice type. Fiction with a human voice AD was assessed by participants as being more interesting and less confusing than fiction with TTS AD. In the case of documentary, the levels of interest and confusion were comparable for both voice types. When asked which voice type they prefer for a given genre, participants showed a preference for AD read by a human for fiction, but no preference between a human voice AD and TTS AD for documentary. Overall, the findings suggest that when properly delivered, both in terms of style and voice, AD may increase the chances of B/VIP audiences having a more engaging viewing experience. It is hoped that this PhD thesis will act as a useful stepping stone towards understanding the links between the AD product and user context, and towards conducting further research in this field.

L'utilisation de réponses impulsionnelles spatiales de salles (spatial room impulse response, SRIR) dans la reproduction d'effets de réverbération de salle tri-dimensionnels connaît aujourd'hui une réelle démocratisation grâce à la commercialisation répandue d'antennes sphériques de microphones (spherical microphone array, SMA) et à une capacité de calcul numérique en croissance continue. Ces SRIR peuvent reproduire des effets de réverbération spatialisés sur des dispositifs immersifs ("surround-sound") à travers des convolutions multicanal de plus en plus performantes. De cette utilisation découle naturellement une demande pour des techniques d'analyse et de traitement non seulement capables d'assurer une reproduction fidèle, mais qui pourraient éventuellement aussi servir à contrôler différentes modifications de la SRIR de façon plus créative que réaliste. Dans ce contexte, l'objectif principal de cette thèse est de développer un environnement complet d'analyse, de traitement, et de manipulation temps-fréquence-espace de SRIR. Les outils d'analyse doivent mener à une modélisation approfondie permettant ensuite un traitement de la mesure vis-à-vis de ses limitations intrinsèques (conditions de mesure, accumulation de bruit de fond, etc.) ainsi qu'une capacité à faire évoluer certaines caractéristiques de l'effet de réverbération décrit par la SRIR. Ces caractéristiques peuvent être tout à fait objectives, c'est-à-dire explicitement reliées à différents paramètres du modèle, ou alors plutôt informées par une connaissance de la perception humaine de l'acoustique des salles. Les aspects théoriques de ce projet de recherche sont présentés en deux parties principales. Dans un premier temps, le modèle de signal de SRIR sous-jacent est décrit en s'inspirant directement des approches historiques dans les domaines de la réverbération artificielle et le traitement de SMA, tout en y proposant plusieurs extensions. Le modèle de signal est alors exploité afin de définir les méthodes d'analyse qui forment le noyau du cadre de traitement-manipulation final. Ces méthodes se focalisent particulièrement sur (a) l'identification du "temps de mélange" décrivant le moment de transition entre les premières réflexions et la réverbération tardive, (b) la génération d'une cartographie temps-espace des premières réflexions, et (c) l'estimation des paramètres régissant la décroissance exponentielle de l'enveloppe d'énergie de la réverbération tardive, à la fois en fréquence et en direction. La définition d'une procédure de génération de représentations directionnelles de SRIR (directional room impulse response, DRIR) est aussi nécessaire pour pouvoir prendre en compte la dépendance directionnelle de ces propriétés. En seconde partie, les paramètres de modélisation explicités par les méthodes d'analyse sont exploités à des fins soit de traitement (c'est-à-dire tenter de corriger certaines des limitations inhérentes au processus de mesure par SMA), soit de manipulation ou de modification plus créative de la SRIR. Deux méthodes de traitement sont développées en particulier : (1) une procedure d'atténuation de bruits non stationnaires agissant directement sur les signaux de mesure par balayages de fréquence exponentiels (exponential sweep method, ESM) répétés, et (2) une technique de débruitage basée sur une extrapolation et une resynthèse de la queue de réverbération tardive. Les descriptions théoriques ainsi complétées, les principales méthodes d'analyse ainsi que la génération de DRIR et le débruitage sont sujets à une série de tests de validation au cours desquels des SRIR simulées sont utilisées afin d'évaluer la performance, les limitations, et la paramétrisation des différentes techniques. Ces sous-études permettent à chaque méthode d'être vérifiée individuellement, et donnent un aperçu détaillé du fonctionnement interne des outils d'analyse. Enfin, une vue d'ensemble de l'environnement d'analyse-traitement-manipulation est obtenue [...]
The use of spatial room impulse responses (SRIR) for the reproduction of three-dimensional reverberation effects through multi-channel convolution over immersive surround-sound loudspeaker systems has become commonplace within the last few years, thanks in large part to the commercial availability of various spherical microphone arrays (SMA) as well as a constant increase in computing power. This use has in turn created a demand for analysis and treatment techniques not only capable of ensuring the faithful reproduction of the measured reverberation effect, but which could also be used to control various modifications of the SRIR in a more "creative" approach, as is often encountered in the production of immersive musical performances and installations. Within this context, the principal objective of the current thesis is the definition of a complete space-time-frequency framework for the analysis, treatment, and manipulation of SRIRs. The analysis tools should lead to an in-depth model allowing for measurements to first be treated with respect to their inherent limitations (measurement conditions, background noise, etc.), as well as offering the ability to modify different characteristics of the final reverberation effect described by the SRIR. These characteristics can be either completely objective, even physical, or otherwise informed by knowledge of human auditory perception with regard to room acoustics. The theoretical work in this research project is therefore presented in two main parts. First, the underlying SRIR signal model is described, heavily inspired by the historical approaches from the fields of artificial reverberation synthesis and SMA signal processing, while at the same time (incrementally) extending both. The signal model is then used to define the analysis methods that form the core of the final framework; these focus particularly on (a) identifying the "mixing time" that defines the moment of transition between the early reflection and late reverberation regimes, (b) obtaining a space-time cartography of the early reflections, and (c) estimating the frequency- and direction-dependent properties of the late reverberation's exponential energy decay envelope. In order to account for the directional dependence of these properties, a procedure for generating directional SRIR representations (i.e. directional room impulse responses, DRIR) that guarantee the preservation of certain fundamental reverberation properties must also be defined. In the second part, the model parameters made explicit by the analysis methods are exploited in order to either treat (i.e. attempt to correct some of the inevitable limitations inherent to the SMA measurement process) or more creatively manipulate and modify the SRIR. Two treatment methods in particular are developed in this thesis: (1) a pre-analysis procedure acting directly on repeated exponential sweep method (ESM) SMA measurement signals in an attempt to simultaneously increase the resulting SRIR's signal-to-noise ratio (SNR) while reducing its vulnerability to non-stationary noise events, and (2) a post-analysis denoising technique based on replacing the SRIR's background noise floor with a resynthesized extrapolation of the late reverberation tail. The theoretical descriptions thus complete, the main analysis methods as well as the DRIR generation and the denoising treatment procedures are then subjected to a series of validation tests, wherein simulated SRIRs (or parts thereof) are used to evaluate the performance, discuss the limitations, and parameterize the implementation of the different techniques. These sub-studies allow each method to be individually verified, resulting in a comprehensive investigation into the inner workings of the analysis toolbox (as well as the denoising process). Finally, to provide a concluding overview of the complete analysis-treatment-manipulation framework, similar studies are carried out using examples of real-world [...]

The development and improvement of 3-D immersive audio is gaining momentum through the growing interest in virtual reality. Possible applications reach from recreating real world environments to immersive concerts and performances to exploiting big data acoustically. To improve the immersive experience several measures can be taken. The recording of the sound field, the spatialization and the development of the loudspeaker arrays are some of the greatest challenges. In this thesis, these challenges for improving immersive audio will be explored. First, there will be a short introduction about 3D audio and a review about the state of the art technology and research. Next, the thesis will provide an introduction to 3D loudspeaker arrays and describe the systems used during this research. Furthermore, the development of a new 16-element 3rd order sound field microphone will be described. Afterwards, different spatial audio algorithms such as higher order ambisonics, wave field synthesis and vector based amplitude panning will be described, analyzed and compared. For each spatialization algorithm, the quality of soundfield reproduction will be quantified using listener perception tests for clarity and sound source localization.
Master of Science

In immersive audio environments with distributed microphones, Independent Component Analysis (ICA) can be applied to uncover signals from a mixture of other signals and noise, such as in a cocktail party recording. ICA algorithms have been developed for instantaneous source mixtures and convolutional source mixtures. While ICA for instantaneous mixtures works when no delays exist between the signals in each mixture, distributed microphone recordings typically result various delays of the signals over the recorded channels. The convolutive ICA algorithm should account for delays; however, it requires many parameters to be set and often has stability issues. This thesis introduces the Channel Aligned FastICA (CAICA), which requires knowledge of the source distance to each microphone, but does not require knowledge of noise sources. Furthermore, the CAICA is combined with Time Frequency Masking (TFM), yielding even better SOI extraction even in low SNR environments. Simulations were conducted for ranking experiments tested the performance of three algorithms: Weighted Beamforming (WB), CAICA, CAICA with TFM. The Closest Microphone (CM) recording is used as a reference for all three. Statistical analyses on the results demonstrated superior performance for the CAICA with TFM. The algorithms were applied to experimental recordings to support the conclusions of the simulations. These techniques can be deployed in mobile platforms, used in surveillance for capturing human speech and potentially adapted to biomedical fields.

The complex relationship between array gain patterns and microphone distributions limits the application of traditional optimization algorithms on irregular arrays, which show enhanced beamforming performance for human speech capture in immersive environments. This work analyzes the relationship between irregular microphone geometries and spatial filtering performance with statistical methods. Novel geometry descriptors are developed to capture the properties of irregular microphone distributions showing their impact on array performance. General guidelines and optimization methods for regular and irregular array design are proposed in immersive (near-field) environments to obtain superior beamforming ability for speech applications. Optimization times are greatly reduced through the objective function rules using performance-based geometric descriptions of microphone distributions that circumvent direct array gain computations over the space of interest. In addition, probabilistic descriptions of acoustic scenes are introduced to incorporate various levels of prior knowledge for the source distribution. To verify the effectiveness of the proposed optimization methods, simulated gain patterns and real SNR results of the optimized arrays are compared to corresponding traditional regular arrays and arrays obtained from direct exhaustive searching methods. Results show large SNR enhancements for the optimized arrays over arbitrary randomly generated arrays and regular arrays, especially at low microphone densities. The rapid convergence and acceptable processing times observed during the experiments establish the feasibility of proposed optimization methods for array geometry design in immersive environments where rapid deployment is required with limited knowledge of the acoustic scene, such as in mobile platforms and audio surveillance applications.

The Generalized Sidelobe Canceller is an adaptive algorithm for optimally estimating the parameters for beamforming, the signal processing technique of combining data from an array of sensors to improve SNR at a point in space. This work focuses on the algorithm’s application to widely-separated microphone arrays with irregular distributions used for human voice capture. Methods are presented for improving the performance of the algorithm’s blocking matrix, a stage that creates a noise reference for elimination, by proposing a stochastic model for amplitude correction and enhanced use of cross correlation for phase correction and time-difference of arrival estimation via a correlation coefficient threshold. This correlation technique is also applied to a multilateration algorithm for an efficient method of explicit target tracking. In addition, the underlying microphone array geometry is studied with parameters and guidelines for evaluation proposed. Finally, an analysis of the stability of the system is performed with respect to its adaptation parameters.

Al giorno d'oggi si registra un grande interesse verso i sistemi di telecon- ferenza multimediale a seguito della crescente richiesta di comunicazioni effi- cienti e dello sviluppo di tecniche avanzate per il processamento digitale dei segnali. Un sistema di teleconferenza dovrebbe fornire una rappresentazione realistica del campo sonoro e visivo, consentendo una comunicazione natu- rale tra i partecipanti dislocati ovunque nel mondo come fossero nella stessa stanza. In questo contesto, sono stati sviluppati molti sistemi, a partire da applicazioni basate su PC pensate per comunicazioni tra singoli utenti no a sistemi complessi dotati di ampi schermi che riproducono la stanza remota come fosse il proseguimento della stanza locale. Nei sistemi di teleconferenza è possibile ridurre l'eco indesiderata dovuta all'accoppiamento tra l'altoparlante e il microfono usando un cancellatore d'eco acustica (AEC). In presenza di più di un partecipante, per la localizza- zione del parlatore devono essere presi in considerazione sistemi multicanale. Possono essere ottenute prestazioni più realistiche già con sistemi stereofonici, poichè gli ascoltatori hanno a disposizione informazioni spaziali che aiutano ad identi care la posizione del parlatore. Tuttavia, deve essere impiegato un maggior numero di ltri adattativi e la relazione lineare esistente tra i due canali generati dalla stessa sorgente causa problemi aggiuntivi: la soluzione dell'algoritmo adattativo non è unica e dipende dalla posizione del parla- tore nella stanza di trasmissione che non è stazionaria, causando possibili problemi di convergenza. In aggiunta, la scelta dell'algoritmo adattativo di- venta estremamente importante perchè le prestazioni dipendono dal numero di condizionamento del segnale d'ingresso che è molto alto nello scenario multicanale. In questa tesi, vengono presentati contributi innovativi per la cancellazione d'eco acustica stereofonica basati sul fenomeno della \missing- fundamental". L'innovazione delle soluzioni è legata alla grande riduzione della coerenza tra i canali del segnale stereo che si riesce ad ottenere senza al- terare la qualità dell'audio e la percezione stereofonica. Inoltre, viene discussa una soluzione per migliorare la velocità di convergenza dei filtri adattativi basata su un metodo per la variazione del passo d'adattamento: l'approccio è applicato alla cancellazione d'eco acustica stereofonica ma in realtà può essere usato per generici algoritmi adattativi. Contestualmente, si è assistito ad un crescente interesse nel progetto di si- stemi che forniscono una riproduzione dei suoni la più realistica possibile così che l'ascoltatore non si accorge che sono stati prodotti artifi cialmente poichè è immerso nella scena audio virtuale circondato da un elevato numero di altoparlanti. I sistemi convezionali sono progettati per massimizzare la senzazione acustica in una speci ca posizione dell'ambiente d'ascolto, il cosiddetto sweet spot. Inoltre, non è possibile ottenere una corretta localiz- zazione della sorgente con un numero limitato di altoparlanti. Quindi, sono stati condotti diversi studi sull'ottimizzazione di questi sistemi, concentrando l'attenzione su nuove tecniche di registrazione e riproduzione, ovvero la Wave Field Analysis (WFA) e la Wave Field Synthesis (WFS). La prima è una tec- nica di registrazione del campo sonoro basata su array di microfoni e la seconda consente la sintesi del campo sonoro attraverso array di altoparlanti. Per utilizzare queste tecniche in scenari reali (ad esempio, sistemi di telecon- ferenza, cinema, home theatre) è necessario applicare algoritmi multicanale per il processamento digitale dei segnali, già sviluppati per sistemi tradizio- nali. Questo porta all'introduzione della Wave Domain Adaptive Filtering (WDAF), ovvero una generalizzazione spazio-temporale dell'algoritmo adat- tativo Fast Least Mean Squares, consentendo una considerevole riduzione della complessità computazionale. In questa tesi vengono discusse soluzioni efficienti per un'implementazione in tempo reale e possibili approssimazioni di fase delle funzioni guida usate per gestire gli altoparlanti. Inoltre, vengono presentati un approccio per la WDAF basato sulla struttura Weighted-Overlap-Add e una tecnica per il puntamento digitale dei arrays lineari basata sulla WFS: l'obiettivo di questi studi è quello di applicare questi concetti in scenari reali, come nel caso di un sistema di teleconferenza. Infatti, le suddette tecniche per la riproduzione audio immersiva possono essere sfruttate per migliorare le prestazioni di si- stemi di teleconferenza a grandezza naturale, combinando requisiti temporali e spaziali. Inoltre, risultano necessari algoritmi di riproduzione audio per migliorare la qualità audio percepita così da rendere più piacevole l'ambiente d'ascolto tenendo conto di alcune caratteristiche proprie dell'ambiente. Più speci ficata- mente, l'equalizzazione rappresenta uno strumento potente capace di gestire le irregolarità della risposta in frequenza: un equalizzatore può compensare il posizionamento del parlatore e le caratteristiche della stanza d'ascolto e può essere applicato in un sistema di teleconferenza per rendere la comunicazione la più naturale possibile. In questo lavoro vengono discusse la valutazione di un equalizzatore multipunto e una soluzione mixed-phase con un ritardo di gruppo della stanza adeguatamente progettato.
Nowadays, there is a large interest towards multimedia teleconferencing sys- tems as a consequence of the increasing requirement for efficent communica- tions and the development of advanced digital signal processing techniques. A teleconferencing system should provide a realistic representation of visual and sound fields, allowing a natural communication among participants any- where in the world as they were all in the same room. In this context, a lot of systems have been developed ranging from PC-based applications, thought for single users communications, up to complex systems provided with large video screens playing the remote room as it were a continuum of the local room. In teleconferencing systems the undesired echo due to coupling between the loudspeaker and the microphone can be reduced using an acoustic echo can- celer (AEC). In the presence of more than one participant, multichannel systems have to be taken into consideration for speaker localization. More realistic performance can be already obtained through stereophonic systems since listeners have spatial information that helps to identify the speaker position. Anyway, more adaptive lters have to be used and the linear rela- tionship existing between the two channels generated from the same source brings some additional problems: the solution of the adaptive algorithm is not unique and depends on the speaker position in the transmission room which is not stationary, causing possible convergence problems. Moreover, the choice of the adaptive algorithm becomes extremely important because the performance depends on the condition number of the input signal which is very high in the multichannel scenario. In this thesis novel contributions for stereophonic acoustic echo cancellation are given based on the \missing- fundamental" phenomenon. The novelty of the solutions is related to the great interchannel coherence reduction obtained without a ecting speech quality and stereo perception. Moreover, a solution for improving the con- vergence speed of adaptive lters is discussed based on a variable step-size method: the approach is applied to stereophonic acoustic echo cancellation but, actually, it can be used for generic adaptive algorithms. Contextually, there has been an increasing interest in the design of systems providing a reproduction of sounds as realistic as possible so that the lis- tener does not notice that they have been produced arti cially since he is immersed in the virtual audio scene surrounded by a large number of loud- speakers. Conventional systems are designed to obtain the optimal acoustic sensation in a particular position of the listening environment, i.e., the so called sweet spot. Furthermore, it is impossible to achieve a correct source localization with a limited number of loudspeakers. Hence, several research e orts have been made in the optimization of these systems, focusing on new recording and reproduction techniques, i.e., Wave Field Analysis (WFA) and Wave Field Synthesis (WFS). The former is a sound eld recording tech- nique based on microphone arrays and the latter allows sound eld synthesis through loudspeakers arrays. At the aim of using these techniques in real world applications (e.g., teleconferencing systems, cinemas, home theatres) it is necessary to apply multichannel digital signal processing algorithms, already developed for traditional systems. This led to the introduction of Wave Domain Adaptive Filtering (WDAF), a spatio-temporal generalization of Fast Least Mean Squares adaptive algorithm, allowing a considerable re- duction of the computational complexity. Efficient solutions for real time implementation and possible phase approx- imations of the driving functions used in order to manage the loudspeakers are discussed in this thesis. Furthermore, a Weighted-Overlap-Add-based (WOLA-based) approach for WDAF and a WFS-based digital pointing of line arrays are presented: the objective of these studies is that of apply- ing these concepts in real scenarios, such as a teleconferencing system. In- deed, the aforementioned immersive audio reproduction techniques can be exploited for enhancing the performance of life-sized teleconferencing sys- tems, combining temporal and spatial requirements. Furthermore, audio rendering algorithms are needed to improve the perceived audio quality in order to make the listening environment more pleasant by taking into account some speci c features of the environment. More specifically, equalization represents a powerful tool capable of dealing with the frequency response irregularities: an equalizer can compensates for speaker placement and listening room characteristics and it can be applied in a tele-conferencing system to make the communication the most natural as possible. The evaluation of a multipoint equalizer and a mixed-phase solution with a suitably designed room group delay are discussed in this work.

Virtual reality applications are on the rise and touching numerous domains, including healthcare, training, and gaming, to name a few. Existing experiences, however, are not fully immersive, as only two senses (audio-visual) are stimulated. To overcome this limitation, olfactory and haptic devices are emerging, thus making multisensory immersive experiences a reality. To date, however, little is known about the impact that each stimulated sense has on the overall experience, as well as on the user’s sense of e.g., realism, immersion, and engagement. In this pilot, we aim to answer this question. Using a multisensory pod, sixteen participants were immersed in a 2.5-minute virtual world, where smells, vibroacoustic, and somatosensory stimuli (i.e., wind and heat) were presented, in addition to 360-degree video and surround sound. Using two wearable devices, we kept track of the user’s heart rate, breathing rate, skin temperature, blood volume pulse, and electrodermal activity while they were immersed. In this paper, we report the impact that stimulating different senses had on the users’ overall experience, sense of presence, immersion, realism, flow, cybersickness, and emotional states, both subjectively, as well as objectively using features extracted from the wearable devices.

As virtual reality technologies emerge, the ability to create immersive experiences visually drastically improved [1]. However, in order to accompany the visual immersion, audio must also become more immersive [2]. This is where 3D audio comes in. 3D audio allows for the simulation of sounds from specific directions, allowing a more realistic feeling [3]. At the present moment, there lacks sufficient tools for users to design immersive audio experiences that fully exploit the abilities of 3D audio. This paper proposes and implements the following systems [4]: 1. Automatic separation of stems from the incoming audio file, or letting the user upload the stems themselves 2. A simulated environment in which the separated stems will be automatically placed in 3. A user interface in order to manipulate the simulated positions of the separated stems. We applied our application to a few selected audio files in order to conduct a qualitative evaluation of our approach. The results show that our approach was able to successfully separate the stems and simulate a dimensional sound effect.

With the introduction of immersive audio formats like MPEG-H and Dolby AC-4, TV terminals may allow end users to configure their experience by changing values of audio parameters directly or indirectly (audio profiles) via some kind of configuration user interface. In order to support more dynamic user interfaces for this kind of interactive audio setup, DTV apps should be able to manipulate such parameters as well. This contribution proposes the addition of immersive audio properties to media elements in NCL applications.

This work, situated at Rensselaer’s Collaborative-Research Augmented Immersive Virtual Environment Laboratory (CRAIVE-Lab), uses panoramic image datasets for spatial audio display. A system is developed for the room-centered immersive virtual reality facility to analyze panoramic images on a segment-by-segment basis, using pre-trained neural network models for semantic segmentation and object detection, thereby generating audio objects with respective spatial locations. These audio objects are then mapped with a series of synthetic and recorded audio datasets and populated within a spatial audio environment as virtual sound sources. The resulting audiovisual outcomes are then displayed using the facility’s human-scale panoramic display, as well as the 128-channel loudspeaker array for wave field synthesis (WFS). Performance evaluation indicates effectiveness for real-time enhancements, with potentials for large-scale expansion and rapid deployment in dynamic immersive virtual environments.

The Spatial Audio Data Immersive Experience (SADIE) project aims to identify new foundational relationships pertaining to hu-man spatial aural perception, and to validate existing relation-ships. Our infrastructure consists of an intuitive interaction in-terface, an immersive exocentric sonification environment, and a layer-based amplitude-panning algorithm. Here we highlight the system’s unique capabilities and provide findings from an initial externally funded study that focuses on the assessment of human aural spatial perception capacity. When compared to the existing body of literature focusing on egocentric spatial perception, our data show that an immersive exocentric environment enhances spatial perception, and that the physical implementation using high density loudspeaker arrays enables significantly improved spatial perception accuracy relative to the egocentric and virtual binaural approaches. The preliminary observations suggest that human spatial aural perception capacity in real-world-like immersive exocentric environments that allow for head and body movement is significantly greater than in egocentric scenarios where head and body movement is restricted. Therefore, in the design of immersive auditory displays, the use of immersive exocentric environments is advised. Further, our data identify a significant gap between physical and virtual human spatial aural perception accuracy, which suggests that further development of virtual aural immersion may be necessary before such an approach may be seen as a viable alternative.

The article analyzes the peculiarities of the media content shaping and transformation in the convergent dimension of cross-media, taking into account the possibilities of augmented reality. With the help of the principles of objectivity, complexity and reliability in scientific research, a number of general scientific and special methods are used: method of analysis, synthesis, generalization, method of monitoring, observation, problem-thematic, typological and discursive methods. According to the form of information presentation, such types of media content as visual, audio, verbal and combined are defined and characterized. The most important in journalism is verbal content, it is the one that carries the main information load. The dynamic development of converged media leads to the dominance of image and video content; the likelihood of increasing the secondary content of the text increases. Given the market situation, the effective information product is a combined content that combines text with images, spreadsheets with video, animation with infographics, etc. Increasing number of new media are using applications and website platforms to interact with recipients. To proceed, the peculiarities of the new content of new media with the involvement of augmented reality are determined. Examples of successful interactive communication between recipients, the leading news agencies and commercial structures are provided. The conditions for effective use of VR / AR-technologies in the media content of new media, the involvement of viewers in changing stories with augmented reality are determined. The so-called immersive effect with the use of VR / AR-technologies involves complete immersion, immersion of the interested audience in the essence of the event being relayed. This interaction can be achieved through different types of VR video interactivity. One of the most important results of using VR content is the spatio-temporal and emotional immersion of viewers in the plot. The recipient turns from an external observer into an internal one; but his constant participation requires that the user preferences are taken into account. Factors such as satisfaction, positive reinforcement, empathy, and value influence the choice of VR / AR content by viewers.