Дисертації з теми "Brain language processing"

It has been shown that music and language training can elicit plastic changes on brain structure and function bringing along behavioural benefits. For instance, musicians have been reported to have better auditory discrimination including pitch and speech-in-noise perception, motor-synchronization, verbal memory and general IQ than individuals without formal musical background. Also, bilinguals have shown higher executive function and attention-related abilities than monolinguals. Furthermore, altered functional and structural connectivity can be tracked to brain areas related to the activities most frequently performed by both musicians (instrumentalists and singers) and linguistic experts (such as bilinguals or professional phoneticians). While research in the last decade has devoted important effort to the study of brain plasticity, only a few investigations have addressed the connection between the initial functional or structural properties of brain networks related to auditory-motor function and subsequent language or musical training. Indeed, brain structural markers such as grey matter volume/density or white-matter diffusivity measurements from diffusion tensor imaging (DTI) data, as well as functional measurements from task- related activity or resting-state data from magnetic resonance imaging (MRI) or electroenceplhalography (EEG) have been demonstrated to correlate with consecutive performance and learning in the auditory-motor domain. The main goal of the present dissertation was twofold: we aimed to further the existing knowledge regarding brain plasticity elicited during putative sensitive periods and after long-term music practice, and to explore the white-matter pathways that predict linguistic or musical skills at baseline . Our secondary goals were to confirm previous findings regarding the brain structures involved in music and language processing, as well as to provide evidence of the benefits of usingstructural measurements and correlational analyses between imaging and behavioural data to study inter-individual differences. Study I focused on the comparison between professional pianists and non- musicians observing a complex pattern of increases and decreases in grey matter volume. In comparison to non-musician individuals, pianists showed greater grey matter volume in areas related to motor skill and the automatization of learned movements, as well as reinforcement learning and emotional processing. On the other hand, regions associated to sensorimotor control, score reading and auditory and musical perception presented a reduction in grey matter volume. Study II explored the relationship between white-matter structural properties of the arcuate fasciculus (AF) and the performance of native German speakers in a foreign- language (Hindi) sentence and word imitation task. We found that a greater left lateralization of the AF volume predicted performance on the imitation task. This result was confirmed by using not only a manual deterministic approach but also an automatic atlas-based fibre-reconstruction method, which in addition pointed out to a specific region in the anterior half of the left AF as the most related to imitation ability. Study III aimed to investigate whether the white-matter structural connectivity of the pathways previously described as targets for plasticity mechanisms in professional musicians predicted musical abilities in non-musicians. We observed that the white- matter microstructural organization of the right hemisphere pathways involved in motor-control (corticospinal tract) and auditory-motor transformations (AF) correlated with the performance of non-musician individuals during the initial stages of rhythmic and melodic learning. The present work confirmed the involvement of several brain structures previously described to display plastic effects associated to music and language training in the first stages of audio-motor learning. Furthermore, they challenge previous views regarding music-induced plasticity by showing that expertise is not always or uniquely correlated with increases in brain tissue. This raises the question of the role of efficiency mechanisms derived from professional-like practice. Most importantly, the results from these three studies converge in showing that a prediction-feedback-feedforward loop for auditory-motor processing may be crucially involved in both musical and language learning and skills. We thus suggest that brain auditory-motor systems previously described as participating in native language processing (cortical areas of the dorsal route for language processing and the AF that connects them) may also be recruited during exposure to new linguistic or musical material, being refined after sustained music practice.
Estudios previos muestran que la formación musical y lingüística provoca cambios plásticos en las estructuras y funciones cerebrales, acompañándose también de beneficios conductuales. Por ejemplo, se ha descrito que los músicos poseen mejores habilidades de discriminación auditiva (incluyendo la percepción tonal y la discriminación del habla en un ambiente ruidoso), una mayor capacidad de sincronización motora, así como mejor memoria verbal y coeficiente intelectual general en comparación con personas sin formación musical. Paralelamente, los bilingües muestran mejores funciones ejecutivas y habilidades relacionadas con la atención en comparación con individuos monolingües. Además, las alteraciones en la conectividad cerebral funcional y estructural pueden ser rastreadas estudiando las áreas cerebrales relacionadas con las actividades más utilizadas por músicos (instrumentistas y cantantes) y expertos lingüísticos (como bilingües o fonetistas profesionales). Pese a que en la última década se han dedicado esfuerzos importantes en el campo de la investigación sobre la plasticidad cerebral, sólo unos pocos estudios han tratado de investigar la conexión entre las propiedades iniciales del cerebro, en cuanto a las funciones y estructuras que se relacionan con las funciones auditivo-motoras, y el posterior aprendizaje musical o del lenguaje. Sin embargo, los marcadores estructurales cerebrales, tales como volumen/densidad de materia gris o medidas de difusividad en la sustancia blanca a partir de datos de imagen del tensor de difusión, así como medidas funcionales de la actividad relacionada con una tarea o datos de resting-state (estado de reposo) obtenidos por resonancia magnética o electroencefalografía, han demostrado que pueden correlacionar con el rendimiento y el aprendizaje en el dominio auditivo- motor. En la presente tesis pretendíamos ampliar nuestro conocimiento en cuanto a la plasticidad cerebral obtenida durante los supuestos “períodos sensibles” y después de la práctica musical mantenida en el tiempo, por un lado, y explorar las vías de sustancia blanca que pueden predecir habilidades lingüísticas o musicales al inicio del aprendizaje, por otro lado. Como objetivos secundarios, queríamos confirmar resultados previos con respecto a las estructuras cerebrales involucradas en el procesamiento de la música y el lenguaje, así como apoyar el uso de mediciones estructurales y enfoques correlacionales (entre datos de neuroimagen y conductuales) para estudiar las diferencias inter- individuales. El Estudio I se centró en la comparación entre pianistas profesionales y no músicos, observando un complejo patrón de aumentos y disminuciones en el volumen de materia gris. En comparación con los individuos no músicos, los pianistas mostraron mayor volumen de sustancia gris en áreas relacionadas con la habilidad motora y la automatización de movimientos aprendidos, así como el aprendizaje a través del refuerzo y el procesamiento emocional, mientras que las regiones asociadas al control sensoriomotor, lectura de partituras y percepción auditiva y musical presentaron una reducción del volumen de materia gris. El Estudio II exploró la relación entre las propiedades estructurales de la materia blanca del fascículo arqueado (AF por sus siglas en inglés) y el rendimiento de hablantes nativos de alemán en una tarea de imitación de frases y palabras en una lengua extranjera (hindi). Encontramos que una mayor lateralización del volumen de AF hacia la izquierda predecía el desempeño en la tarea de imitación. Este resultado se confirmó utilizando no sólo un enfoque determinístico-manual sino también una reconstrucción automática (basada en atlas anatómicos) de las fibras de sustancia blanca que, además, señalaba una región específica en la mitad anterior del AF izquierdo como la más relacionada con las capacidades de imitación. El Estudio III tenía como objetivo investigar si la conectividad estructural de vías de sustancia blanca anteriormente descritas como dianas para los mecanismos de plasticidad en músicos profesionales, podría predecir las habilidades musicales en los no músicos. Se observó que la organización micro-estructural de la materia blanca en el hemisferio derecho en vías involucradas en el control motor (tracto corticoespinal) y en transformaciones auditivo-motoras (AF) correlacionaba con el desempeño de individuos no músicos en las etapas iniciales del aprendizaje rítmico y melódico. El presente trabajo ha confirmado la implicación en las primeras etapas del aprendizaje audio-motor de varias estructuras cerebrales que previamente habían mostrado efectos plásticos asociados al aprendizaje musical y del lenguaje. Además, estos resultados desafían las opiniones anteriores sobre la plasticidad inducida por la experiencia musical al demostrar que la experiencia no se correlaciona siempre ni únicamente con un aumento del tejido cerebral, y planteando así preguntas sobre los mecanismos de eficiencia derivados de la práctica musical a nivel profesional. Más importante aún es que los resultados de estos tres estudios convergen mostrando que un bucle de predicción–retroalimentación (feedback)–alimentación directa (feedforward) para el procesamiento auditivo-motor puede estar implicado de manera crucial tanto en el aprendizaje musical como en el aprendizaje de idiomas. Por tanto, sugerimos que los sistemas auditivo-motrices del cerebro, que previamente se habían descrito como participantes en el procesamiento del lenguaje nativo (áreas corticales involucradas en la vía dorsal para el procesamiento del lenguaje, y el AF, que las conecta) también pueden ser reclutados durante la exposición a material lingüístico o musical nuevo, siendo refinado tras años de práctica musical activa.

Thesis (Ph. D.)--University of Oregon, 2004.
Typescript. Includes vita and abstract. Includes bibliographical references (leaves 142-166). Also available for download via the World Wide Web; free to University of Oregon users.

Humans display distinct unlimited capacity to produce expressions in language and use them flexibly in language processing. This characteristic of human language allows speakers to use novel, flexible, and complex structures during communication. Neurobiologically, however, it is not fully understood how the rapid process of language production and language comprehension occurs, including word generation, interpretations and common representations that facilitate the process of real-time language processing. The classical theories and approaches have limited the language network to perisylvian cortical regions, namely the Broca’s and Wernicke’s areas. This thesis proposes that the language network goes beyond the cortical regions indicated by traditional views. In doing so, this thesis puts forward a hypothesis that subcortical structures are not only fundamental to memory but also to language, in which online language processing receives a major contribution from the hippocampal declarative memory, which allows speakers and listeners to use language flexibly. The mechanism of such a contribution by the hippocampal declarative memory system during online language processing is via relational binding in which hippocampal declarative memory rapidly retrieves a network of relative, stored information to serve in the particular context. To support the hypothesis of hippocampal implications in language processing, several pathologies that affect the hippocampus have been reviewed, including Alzheimer’s disease, Down syndrome, Williams’ syndrome, schizophrenia, depression and bipolar disorder. The review evaluated hippocampal neurobiological alterations in each pathology, and determined cognitive and language profiles. Findings from previous pathologies indicate that the hippocampus affects language at two levels. First, in the general delay in language acquisition and other cognitive aspects, and second, in the disturbed use of language during online communication; short lag interaction is seen to occur when the hippocampal formation is lesioned. It appears that hippocampal lesions suppress the flexible use of stored information within certain contexts in communication, as it does in flexible navigation in animal models. This thesis concludes that the hippocampus is a multi-cognitive operator that is implicated in several cognitive areas including the flexible use of language during real-time processing, and therefore it should be taken into account in the language network in the human brain.
Los seres humanos ponen en juego distintas e ilimitadas capacidades para producir expresiones verbales y usarlas de forma flexible durante el procesamiento del lenguaje. Esta característica del lenguaje humano permite al hablante usar estructuras nuevas, flexibles y complejas durante la comunicación. Desde un punto de vista neurobiológico, no se conoce completamente cómo se generan los rápidos procesos de la producción y de la comprensión del lenguaje, incluyendo la generación de palabras, las interpretaciones y las representaciones comunes que facilitan el procesamiento del lenguaje en tiempo real. Las teorías clásicas se han limitado a la red lingüística en las regiones de la corteza perisilviana, es decir, en las áreas de Broca y de Wernicke. La propuesta de esta tesis sugiere que la red lingüística debe ir más allá de las regiones corticales señaladas por las visiones tradicionales. Para hacer esto, la tesis propone la hipótesis de que las estructuras subcorticales no son solo fundamentales para la memoria sino también para el lenguaje, de manera que el procesamiento verbal online recibe una significativa controbución de la memoria declarativa hipocampal que permite a los hablantes y a los oyentes utitlizar flexiblemente el lenguaje. El mecanismo de esta contribución del sistema mnésico declarativo hipocampal durante el procesamiento del lenguaje on-line es a través de la vinculación relacional en la cual la memoria declarativa hipocampal recupera rápidamente una red de información relativa almacenada para ser utilizada en un cierto contexto. Para apoyar la hipótesis de las implicaciones hipocampales en el procesamiento verbal, se han revisado diversas patologías que afectan el hipocampo, incluyendo la enfermedad de Alzheimer, el síndrome de Down, el síndrome de Williams, la esquizofrenia, la depresión y el trastorno bipolar. La revisión ha tenido como objetivo evaluar las alteraciones neurobiológicas hipocampales en cada patología, así como los perfiles cognitivo y lingüístico. Algunos descrubrimientos de patologías previas indican que el hipocampo afecta al lenguaje en dos niveles. Primero, en el retraso general en la adquisición del lenguaje y en otras aspectos cognitivos, y, segundo, en la observación, cuando la formación hipocampal está dañada, del uso alterado del lenguaje durante la comunicación on-line y de la interacción con lapsos cortos. Se observa que las lesiones hipocampales suprimen el uso flexible de información almacenada dentro de un cierto contexto comunicativo, como se ha validado en la navegación flexible en modelos animales. La tesis concluye que el hipocampo es un operador multi-cognitivo que implica a diversas áreas cognitivas que están implicadas en el uso flexible del lenguaje durante el procesamiento en tiempo real, lo que debería tenerse en cuenta en la red del lenguaje en el cerebro humano.

Thesis (Ph. D.)--University of Oregon, 2004.
Typescript. Includes vita and abstract. Includes bibliographical references (leaves 432-460). Also available for download via the World Wide Web; free to University of Oregon users.

Algorithmes et cerveau, bien que de nature extrêmement différentes, sont deux systèmes capables d'effectuer des tâches de langage complexes. En particulier, de récentes avancées en intelligence artificielle ont permis l'émergence d'algorithmes produisant des textes de qualité remarquablement similaire à ceux des humains (ChatGPT, GPT-3). De telles similarités interrogent sur la façon dont le cerveau et ces algorithmes traitent le langage, les mécanismes qu'ils utilisent et les représentations internes qu'ils construisent. Ma thèse consiste à comparer les représentations internes de ces deux systèmes, d'identifier leurs similitudes et leurs différences.Pour ce faire, nous analysons les enregistrements par imagerie fonctionnelle (fMRI) et magnéto-encéphalographie (MEG) de participants écoutant et lisant des histoires, et les comparons aux activations de milliers d'algorithmes de langage correspondant à ces mêmes histoires.Nos résultats mettent d'abord en évidence des similarités de haut niveau entre les représentations internes du cerveau et des modèles de langage. Dans une première partie, nous montrons que les activations des réseaux profonds prédisent linéairement l'activité cérébrale de sujets chez différents groupes (>500 participants), pour différentes modalités d'enregistrement (MEG et fMRI), modalités de stimulus (présentation auditive et visuelle), types de stimulus (mots isolés, phrases et histoires naturelles), langues (néerlandais et anglais) et modèles de langage. Cette correspondance est maximale dans les régions cérébrales souvent associées au langage, pour les algorithmes les plus performants et pour les participants qui comprennent le mieux les histoires. De plus, nous mettons en évidence une hiérarchie de traitement similaire entre les deux systèmes. Les premières couches des algorithmes sont alignées sur les régions de traitement de bas niveau dans le cerveau, telles que les zones auditives et le lobe temporal, tandis que les couches profondes sont alignées sur des régions associées à un traitement de plus haut niveau, notamment les zones fronto-pariétales.Nous montrons ensuite, dans une seconde partie, comment de telles similarités peuvent aider à construire de meilleurs modèles prédictifs de l'activité cérébrale, et à décomposer plus finement dans le cerveau différents processus linguistiques tels que la syntaxe et la sémantique.Enfin, dans une troisième partie, nous explorons les différences entre cerveau et algorithmes. Nous montrons que le cerveau prédit des représentations distantes et hiérarchiques, contrairement aux modèles de langage actuels qui sont principalement entraînés à faire des prédictions à court terme et au niveau du mot. Dans l'ensemble, les algorithmes modernes sont encore loin de traiter le langage de la même manière que les humains le font. Cependant, les liens directs entre leur fonctionnement interne et celui du cerveau fournissent une plateforme prometteuse pour mieux comprendre les deux systèmes, et ouvre la voie à la construction d'algorithmes plus similaires au cerveau
Recent deep language models -- like GPT-3 and ChatGPT -- are capable to produce text that closely resembles that of humans. Such similarity raises questions about how the brain and deep models process language, the mechanisms they use, and the internal representations they construct. In this thesis, I compare the internal representations of the brain and deep language models, with the goal of identifying their similarities and differences. To this aim, I analyze functional resonance imaging (fMRI) and magnetoencephalography (MEG) recordings of participants listening to and reading sentences, and compare them to the activations of thousands of language algorithms corresponding to these same sentences.Our results first highlight high-level similarities between the internal representations of the brain and deep language models. We find that deep nets' activations significantly predict brain activity across subjects for different cohorts (>500 participants), recording modalities (MEG and fMRI), stimulus types (isolated words, sentences, and natural stories), stimulus modalities (auditory and visual presentation), languages (Dutch, English and French), and deep language models. This alignment is maximal in brain regions repeatedly associated with language, for the best-performing algorithms and for participants who best understand the stories. Critically, we evidence a similar processing hierarchy between the two systems. The first layers of the algorithms align with low-level processing regions in the brain, such as auditory areas and the temporal lobe, while the deep layers align with regions associated with higher-level processing, such fronto-parietal areas.We then show how such similarities can be leveraged to build better predictive models of brain activity and better decompose several linguistic processes in the brain, such as syntax and semantics. Finally, we explore the differences between deep language models and the brain's activations. We find that the brain predicts distant and hierarchical representations, unlike current language models that are mostly trained to make short-term and word-level predictions. Overall, modern algorithms are still far from processing language in the same way that humans do. However, the direct links between their inner workings and that of the brain provide an promising platform for better understanding both systems, and pave the way for building better algorithms inspired by the human brain

This study investigated the neural bases underlying representation of nouns and verbs at the semantic, lexical form, and morpho-syntactic levels in Mandarin Chinese, a language with little inflectional morphology. Compared with other studies employing European languages with rich inflections, examination of Chinese would allow the separation of conceptual and morpho-syntactic operations based on different stimulus formats and experimental paradigms. To deal with both the theoretical and design issues in previous studies, several additional measures were taken. First, at each cognitive level, two experiments, one receptive and one expressive, were conducted. Moreover, convergence across experiments at the same cognitive level was computed in order to search for taskindependent grammatical class effects. Second, both concrete and abstract nouns and verbs were included, and conjunction analyses across the two concreteness levels were employed to ensure the generalizability of the findings to all nouns and verbs. Results revealed greater activation for verbs in the left posterior lateral temporal gyri in experiments at both semantic and morpho-syntactic levels, and stronger responses in the prefrontal cortex, including left BA47 and the supplementary motor area, only for morpho-syntactic processing associated with nominal grammatical morphemes, namely, classifiers. No differential levels of activation for nouns and verbs were observed in tasks emphasizing word form representation. While greater activation for processing of nominal classifiers in prefrontal areas may reflect differences in computational complexity associated with selection of grammatical morphemes, the involvement of left posterior lateral temporal cortex has been interpreted as reflecting semantic processing of verbs. The nature of processes represented in each of these regions was further discussed with findings from previous relevant studies. Finally, future studies are proposed for further exploration into the neural mechanisms underlying presentation of nouns and verbs using more recently developed methods of analyses.
published_or_final_version
Speech and Hearing Sciences
Doctoral
Doctor of Philosophy

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Brain and Cognitive Sciences, 2007.
Includes bibliographical references (leaves 139-145).
This thesis examines two questions concerning the working memory system underlying language processing: (1) To what extent is the working memory system underlying language processing domain-specific? and (2) What is the relationship between the working memory system and the long-term memory system in language processing? In Chapter 1, I describe ten experiments investigating the extent to which the working memory system underlying linguistic integrations is domain-specific. I argue that the results of these experiments demonstrate that at least some aspects of the working memory system used for linguistic integrations are not domain-specific, being involved in arithmetic, and possibly, musical processing. In Chapter 2, I describe six experiments investigating the relationship between the two retrieval operations that are required when an incoming word is integrated into an evolving structure: the retrieval of the lexical properties of the word from long-term memory and the retrieval of its structural dependents from working memory. I demonstrate that the relative ease or difficulty of retrieving the lexical properties of an incoming word affect the difficulty of retrieving its structural dependents. I therefore argue that the two retrieval operations rely on overlapping pools of resources.
by Evelina G. Fedorenko.
Ph.D.

Computing is entering an era of hundreds to thousands of processing elements per chip, yet no known parallelism form scales to that degree. To address this problem, we investigate the foundation of a computer architecture where processing elements and memory are contextually partitioned based upon facets of a userâ s life. Such Contextual Partitioning (CP), the situational handling of inputs, employs a method for allocating resources, novel from approaches used in todayâ s architectures. Instead of focusing components on mutually exclusive parts of a task, as in Thread Level Parallelism, CP assigns different physical components to different versions of the same task, defining versions by contextual distinctions in device usage. Thus, application data is processed differently based on the situation of the user. Further, partitions may be user specific, leading to personalized architectures. Our focus is mobile devices, which are, or can be, personalized to one owner. Our investigation is centered on leveraging CP for accurate and real-time speech recognition on mobile devices, scalable to large vocabularies, a highly desired application for future user interfaces. By contextually partitioning a vocabulary, training partitions as separate acoustic models with SPHINX, we demonstrate a maximum error reduction of 61% compared to a unified approach. CP also allows for systems robust to changes in vocabulary, requiring up to 97% less training when updating old vocabulary entries with new words, and incurring fewer errors from the replacement. Finally, CP has the potential to scale nearly linearly with increasing core counts, offering architectures effective with future processor designs.
Master of Science

Nous avons examiné comment des lésions cérébrales chez l'homme affectent les interdépendances entre trois modules de traitement cortical de la couleur, notamment : la perception, la dénomination et la connaissance de la couleur de l'objet. Nous avons étudié la catégorisation des couleurs - un exemple de l'interaction entre la perception et le langage. Les preuves provenant du développement cognitif, de la psychologie comparée et des neurosciences cognitives suggèrent que la catégorisation des couleurs ne provient ni de la perception ni du langage, comme le présume le débat Nature-Nurture. Les catégories de couleurs peuvent plutôt refléter des objets pertinents dans l'environnement. Pour évaluer la causalité entre la catégorisation et la dénomination, nous avons étudié un patient victime d'un AVC, le RDS. Malgré de difficultés pour nommer les couleurs chromatiques, résultant d’une lésion occipito-temporale gauche, la catégorisation des couleurs était relativement épargnée chez RDS. Des expériences d'IRM multimodale ont révélé que la connectivité perception-langage est essentielle pour un nommage efficace des couleurs, mais pas pour une catégorisation. L’étude de la connaissance de la couleur des objets dans le contexte de la dégradation des noms de couleurs de RDS a montré que celui-ci ne pouvait pas lier la perception des couleurs au langage ni à la connaissance sémantique. Il ne pouvait pas associer une couleur visuelle à un nom de couleur ou à la forme de son objet typique. Globalement, nous avons démontré trois ségrégations fonctionnelles dans le traitement cortical des couleurs, entre : (1) la catégorisation et la dénomination des couleurs, (2) la dénomination des couleurs chromatiques et achromatiques et (3) la connaissance des objets colorés et celle des couleurs abstraites. Les mécanismes corticaux du traitement des couleurs pourraient avoir pour objectif de lier des informations sensorielles et sémantiques afin de guider le comportement lié aux objets, en réalisant (1) une perception des couleurs stable, (2) des catégories de couleurs pertinentes et (3) des représentations mentales communes des formes et des couleurs. L'évolution culturelle peut avoir recyclé les circuits neuronaux nécessaires à ces processus pour isoler la couleur de l'objet et l'étiqueter avec des noms de couleur
We examined the effects of brain lesions in humans on the interdependences between three modules of cortical colour processing, namely colour perception, naming and object-colour knowledge. We first focused on colour categorisation - a case-in-point of the interplay between perception and language. Reviewed evidence from cognitive development, comparative psychology and cognitive neuroscience hints that colour categorisation originates from neither perception nor language, as assumed by the Nature-Nurture debate. Instead, colour categories may reflect relevant objects in the environment. To assess the causal link between categorization and naming, we investigated a stroke patient, RDS. Despite severe difficulties in naming chromatic colours, due to a left occipito-temporal lesion, RDS’s colour categorisation was relatively spared. Multimodal MRI experiments revealed that the language-perception connectivity is essential for efficient colour naming but not for categorisation. Investigation of object-colour knowledge in the context of RDS’s colour-naming impairment showed that RDS could not link colour perception to neither language nor semantic knowledge. He could not associate a visual colour to a colour name or to the shape of its typical object. Overall, we demonstrated three functional segregations in colour processing: between (1) colour categorisation and colour naming, (2) naming of chromatic and achromatic colours and (3) knowing about coloured objects and knowing about abstract colours. The main purpose of high-level cortical colour mechanisms could be providing sensory and semantic information to guide object-related behaviour, by achieving (1) stable colour perception, (2) relevant colour categories, and (3) joint mental representations of shapes and colours. These neural computations may have been recycled in cultural evolution to isolate colours from objects and label them with names

How is information organized in the brain during natural reading? Where and when do the required processes occur, such as the perception of individual words and the construction of sentence meanings. How are semantics, syntax and higher-level narrative structure represented? Answering these questions is core to understanding how the brain processes language and organizes complex information. However, due to the complexity of language processing, most brain imaging studies focus only on one of these questions using highly controlled stimuli which may not generalize beyond the experimental setting. This thesis proposes an alternative framework to study language processing. We acquire data using a naturalistic reading paradigm, annotate the presented text using natural language processing tools and predict brain activity with machine learning techniques. Finally, statistical testing is used to form rigorous conclusions. We also suggest the use of direct non-parametric hypothesis tests that do not rely on any model assumptions, and therefore do not suffer from model misspecification. Using our framework, we construct a brain reading map from functional magnetic resonance imaging data of subjects reading a chapter of a popular book. This map represents regions that our model reveals to be representing syntactic, semantic, visual and narrative information. Using this single experiment, our approach replicates many results from a wide range of classical studies that each focus on one aspect of language processing. We extend our brain reading map to include temporal dynamics as well as spatial information by using magnetoencephalography. We obtain a spatio-temporal picture of how successive words are processed by the brain. We show the progressive perception of each word in a posterior to anterior fashion. For each region along this pathway we show a differentiation of the word properties that best explain its activity.

It is widely accepted that the left hemisphere of the brain is specialised and dominant for language comprehension and production and that those with left hemisphere damage often display profound language disruption (Geschwind, 1965). The importance of the left hemisphere is shown by communication problems or extreme difficulty in producing speech following damage to this brain region. In contrast, following right hemisphere damage, disruption to language is less perceptible to the casual observer. The evidence base currently available acknowledges a critical role for the right hemisphere in processing inferred or implied information by maintaining relevant facts and/or suppressing irrelevant ones but the exact role of the right hemisphere and its coordination with the left is open for debate (Johns, Tooley and Traxler, 2008). Two theories have been proposed to explain communication/language difficulties in individuals with right hemisphere damage: (i) the “coarse semantic coding” hypothesis and (ii) the “suppression deficit” hypothesis. The “coarse semantic coding” hypothesis proposes that damage to the right hemisphere causes an over reliance on fine coding assumed to be undertaken by the left hemisphere in the comprehension of language, implying the recall of most literal interpretations. The “suppression deficit” hypothesis proposes that damage in the right hemisphere means multiple activations of meanings of words are not attenuated leading to ineffective suppression of inappropriate interpretations. This project investigated competing evidence for each of these hypotheses by studying the processing abilities of individuals with depressed unilateral brain function caused by stroke or innovatively produced by transcranial DC stimulation (tDCS), on semantic judgement tasks using metaphorical language. The results demonstrated the strongest of evidence for the coarse semantic coding hypothesis when the data from participants with damage to the right hemisphere, both caused by stroke and simulated by tDCS was considered. Overall, the study has furthered the understanding of the role of the right hemisphere in language comprehension and demonstrated the contribution of the tDCS methodology in the field.

Human infants have been shown to have an innate capacity to acquire their mother tongue. In recent decades, the advent of the functional magnetic resonance imaging (fMRI) technique has made it feasible to explore the neural basis underlying language acquisition and processing in children, even in newborn infants (for reviews, see Kuhl & Rivera-Gaxiola, 2008; Kuhl, 2010) . Spontaneous low-frequency (< 0.1 Hz) fluctuations (LFFs) in the resting brain have been shown to be physiologically meaningful in the seminal study (Biswal et al., 1995) . Compared to task-based fMRI, resting-state fMRI (rs-fMRI) has some unique advantages in neuroimaging research, especially in obtaining data from pediatric and clinical populations. Moreover, it enables us to characterize the functional organization of the brain in a systematic manner in the absence of explicit tasks. Among brain systems, the language network has been well investigated by analyzing LFFs in the resting brain. This thesis attempts to investigate the functional connectivity within the language network in typically developing preschool children and the covariation of this connectivity with children’s language development by using the rs-fMRI technique. The first study (see Chapter 2.1; Xiao et al., 2016a) revealed connectivity differences in language-related regions between 5-year-olds and adults, and demonstrated distinct correlation patterns between functional connections within the language network and sentence comprehension performance in children. The results showed a left fronto-temporal connection for processing syntactically more complex sentences, suggesting that this connection is already in place at age 5 when it is needed for complex sentence comprehension, even though the whole functional network is still immature. In the second study (see Chapter 2.2; Xiao et al., 2016b), sentence comprehension performance and rs-fMRI data were obtained from a cohort of children at age 5 and a one-year follow-up. This study examined the changes in functional connectivity in the developing brain and their relation to the development of language abilities. The findings showed that the development of intrinsic functional connectivity in preschool children over the course of one year is clearly observable and individual differences in this development are related to the advancement in sentence comprehension ability with age. In summary, the present thesis provides new insights into the relationship between intrinsic functional connectivity in the brain and language processing, as well as between the changes in intrinsic functional connectivity and concurrent language development in preschool children. Moreover, it allows for a better understanding of the neural mechanisms underlying language processing and the advancement of language abilities in the developing brain.

This dissertation examines the role of language directionality in conference interpret-ing in South Africa with the purpose of highlighting the need for bidirectional inter-preting in a South African context. Western mainstream doctrines prescribe for in-terpreters to work only towards their native (mother) tongue. However, in the multi-lingual South African context, where the majority of the population speaks at least two languages, the idea of language directionality has not been given much thought. Nevertheless, there seems to be a demand for bidirectional interpreters. The study aims to argue in favour of a language bi-directionality in interpreting through empha-sising its theoretical plausibility and practical evidence within the field of conference interpreting in South Africa. The dissertation is the result of an extended literature review and a survey carried out in South Africa in 2013 among conference interpreters working in the country.
Mini-dissertation (MA)--University of Pretoria, 2014.
tm2015
Modern European Languages
MA
Unrestricted

Computers are used for diagnostic and training in the neuropsychological rehabilitation. PLANTEST is a program for the IBM-PC that was developed for diagnostic support. It implements a test that gives information about the reduced ability of brain-injured patients to make plans regarding a certain task.The presented thesis describes a knowledge-based system that can be used to develop new test versions for PLANTEST. The program is called SolvePT and it can prove the solubility of test material used in PLANTEST. It can also automatically generate new test material. The program uses an exhaustive forward-chaining, depth-first search and is implemented in Prolog. The datastructures and algorithm of the program as well as space and time requirements are discussed.
Department of Computer Science

Human infants have been shown to have an innate capacity to acquire their mother tongue. In recent decades, the advent of the functional magnetic resonance imaging (fMRI) technique has made it feasible to explore the neural basis underlying language acquisition and processing in children, even in newborn infants (for reviews, see Kuhl & Rivera-Gaxiola, 2008; Kuhl, 2010) . Spontaneous low-frequency (< 0.1 Hz) fluctuations (LFFs) in the resting brain have been shown to be physiologically meaningful in the seminal study (Biswal et al., 1995) . Compared to task-based fMRI, resting-state fMRI (rs-fMRI) has some unique advantages in neuroimaging research, especially in obtaining data from pediatric and clinical populations. Moreover, it enables us to characterize the functional organization of the brain in a systematic manner in the absence of explicit tasks. Among brain systems, the language network has been well investigated by analyzing LFFs in the resting brain. This thesis attempts to investigate the functional connectivity within the language network in typically developing preschool children and the covariation of this connectivity with children’s language development by using the rs-fMRI technique. The first study (see Chapter 2.1; Xiao et al., 2016a) revealed connectivity differences in language-related regions between 5-year-olds and adults, and demonstrated distinct correlation patterns between functional connections within the language network and sentence comprehension performance in children. The results showed a left fronto-temporal connection for processing syntactically more complex sentences, suggesting that this connection is already in place at age 5 when it is needed for complex sentence comprehension, even though the whole functional network is still immature. In the second study (see Chapter 2.2; Xiao et al., 2016b), sentence comprehension performance and rs-fMRI data were obtained from a cohort of children at age 5 and a one-year follow-up. This study examined the changes in functional connectivity in the developing brain and their relation to the development of language abilities. The findings showed that the development of intrinsic functional connectivity in preschool children over the course of one year is clearly observable and individual differences in this development are related to the advancement in sentence comprehension ability with age. In summary, the present thesis provides new insights into the relationship between intrinsic functional connectivity in the brain and language processing, as well as between the changes in intrinsic functional connectivity and concurrent language development in preschool children. Moreover, it allows for a better understanding of the neural mechanisms underlying language processing and the advancement of language abilities in the developing brain.

Though the current understanding of language processing is incomplete, it has been established that the left hemisphere is dominant for language in the majority of the population. Damage to language centers of the brain and to white matter tracts connecting these language centers results in a language deficit known as aphasia. Neuroplasticity in the brain can often compensate for these language deficits by strengthening neuronal connections between the right and left hemisphere, or by enhancing the neuronal connectivity of undamaged areas in the left hemisphere. Thus the brain can compensate for damaged language centers by using alternative cortical areas. These compensatory language areas may be homologous areas of the right hemisphere, or other undamaged portions of the left hemisphere. Various imaging techniques have been used to demonstrate this phenomenon. The current neuroimaging technique known as quantitative electroencephalographic brain imaging allows investigators to evaluate the functional anatomical location of language processing. When this mapping is overlaid on a magnetic resonance image, investigators are able to locate areas in the brain of the participant that are electrically activated during elicited speech tasks. This method was used in a single case study to examine the brain of an individual with a unique traumatic brain injury in which the anterior portion of the individual's left temporal lobe was surgically removed and considerable recovery of language subsequently occurred. The stimulus for the quantitative electroencephalography included identifying syntactically incorrect sentences. Imaging results from the participant with traumatic brain injury were compared to imaging results obtained from an age-matched control. Differences in quantitative electroencephalography between the two participants included a delayed P1-N1-P2 response and an absent P600 in the participant with traumatic brain injury. Behavioral results include an increased number of incorrect responses from the participant with traumatic brain injury as compared to the control participant. These results imply an interesting cortical distribution of language processing that could be further assessed by functional magnetic resonance imaging.

Abstract Audio-oral speech and visuo-manual sign language as used by the Deaf community are two very different realizations of the human linguistic communication system. Sign language is not only used by the hearing impaired but also by different groups of hearing individuals. To date, there is a great discrepancy in scientific knowledge about signed and spoken languages. Particularly little is known about the integration of the two systems, even though the vast majority of deaf and hearing signers also have a command of some form of speech. This neurolinguistic study aimed to achieve basic knowledge about semantic integration mechanisms across speech and sign language in hearing native and non-native signers. Basic principles of sign processing as reflected in electrocortical brain activation and behavioral decisions were examined in three groups of study participants: Hearing native signers (children of deaf adults, CODAs), hearing late learned signers (professional sign language interpreters), and hearing non-signing controls. Event-related brain potentials (ERPs) and behavioral response frequencies were recorded while the participants performed a semantic decision task for priming lexeme pairs. The lexeme pairs were presented either within speech (spoken prime-spoken target) or across speech and sign language (spoken prime-signed target). Target-related ERP responses were subjected to temporal principal component analyses (tPCA). The neurocognitive basis of semantic integration processes were assessed by analyzing different ERP components (N170, N400, late positive complex) in response to the antonymic and unrelated targets. Behavioral decision sensitivity to the target lexemes is discussed in relation to the measured brain activity. Behaviorally, all three groups of study participants performed above chance level when making semantic decisions about the primed targets. Different result patterns, however, hinted at three different processing strategies. As the target-locked electrophysiological data was analyzed by PCA, for the first time in the context of sign language processing, objectively allocated ERP components of interest could be explored. A little surprisingly, the overall study results from the sign-naïve control group showed that they performed in a more content-guided way than expected. This suggested that even non-experts in the field of sign language were equipped with basic skills to process the cross-linguistically primed signs. Behavioral and electrophysiological study results together further brought up qualitative differences in processing between the native and late learned signers, which raised the question: can a unitary model of sign processing do justice to different groups of sign language users?
Tiivistelmä Kuuloaistiin ja ääntöelimistön motoriikkaan perustuva puhe ja kuurojen yhteisön käyttämä, näköaistiin ja käsien liikkeisiin perustuva viittomakieli ovat kaksi varsin erilaista ihmisen kielellisen viestintäjärjestelmän toteutumismuotoa. Viittomakieltä käyttävät kuulovammaisten ohella myös monet kuulevat ihmisryhmät. Tähänastinen tutkimustiedon määrä viittomakielistä ja puhutuista kielistä eroaa huomattavasti. Erityisen vähän on tiedetty näiden kahden järjestelmän yhdistämisestä, vaikka valtaosa kuuroista ja kuulevista viittomakielen käyttäjistä hallitsee myös puheen jossain muodossa. Tämän neurolingvistisen tutkimuksen tarkoituksena oli hankkia perustietoja puheen ja viittomakielen välisistä semanttisista yhdistämismekanismeista kuulevilla, viittomakieltä äidinkielenään tai muuna kielenä käyttävillä henkilöillä. Viittomien prosessoinnin perusperiaatteita, jotka ilmenevät aivojen sähköisen toiminnan muutoksina ja valintapäätöksinä, tutkittiin kolmessa koehenkilöryhmässä: kuulevilla viittomakieltä äidinkielenään käyttävillä henkilöillä (kuurojen aikuisten kuulevilla ns. CODA-lapsilla, engl. children of deaf adults), kuulevilla viittomakielen myöhemmin oppineilla henkilöillä (viittomakielen ammattitulkeilla) sekä kuulevilla viittomakieltä osaamattomilla verrokkihenkilöillä. Tapahtumasidonnaiset herätepotentiaalit (ERP:t) ja käyttäytymisvasteen frekvenssit rekisteröitiin koehenkilöiden tehdessä semanttisia valintoja viritetyistä (engl. primed) lekseemipareista. Lekseemiparit esitettiin joko puheena (puhuttu viritesana – puhuttu kohdesana) tai puheen ja viittomakielen välillä (puhuttu viritesana – viitottu kohdesana). Kohdesidonnaisille ERP-vasteille tehtiin temporaaliset pääkomponenttianalyysit (tPCA). Semanttisten yhdistämisprosessien neurokognitiivista perustaa arvioitiin analysoimalla erilaisia ERP-komponentteja (N170, N400, myöhäinen positiivinen kompleksi) vastineina antonyymisiin ja toisiinsa liittymättömiin kohteisiin. Käyttäytymispäätöksen herkkyyttä kohdelekseemeille tarkastellaan suhteessa mitattuun aivojen aktiviteettiin. Käyttäytymisen osalta kaikki kolme koehenkilöryhmää suoriutuivat satunnaistasoa paremmin tehdessään semanttisia valintoja viritetyistä kohdelekseemeistä. Erilaiset tulosmallit viittaavat kuitenkin kolmeen erilaiseen prosessointistrategiaan. Kun kohdelukittua elektrofysiologista dataa analysoitiin pääkomponenttianalyysin avulla ensimmäistä kertaa viittomakielen prosessoinnin yhteydessä, voitiin tutkia tarkkaavaisuuden objektiivisesti allokoituja ERP-komponentteja. Oli jossain määrin yllättävää, että viittomakielellisesti natiivin verrokkiryhmän tulokset osoittivat sen jäsenten toimivan odotettua sisältölähtöisemmin. Tämä viittaa siihen, että viittomakieleen perehtymättömilläkin henkilöillä on perustaidot lingvistisesti ristiin viritettyjen viittomien prosessointiin. Yhdessä käyttäytymisperäiset ja elektrofysiologiset tutkimustulokset toivat esiin laadullisia eroja prosessoinnissa viittomakieltä äidinkielenään puhuvien henkilöiden ja kielen myöhemmin oppineiden henkilöiden välillä. Tämä puolestaan johtaa kysymykseen, voiko yksi viittomien prosessointimalli soveltua erilaisille viittomakielen käyttäjäryhmille?

Cette thèse explore la synergie entre l'intelligence artificielle (IA) et la neuroscience cognitive pour faire progresser les capacités de traitement du langage. Elle s'appuie sur l'idée que les avancées en IA, telles que les réseaux neuronaux convolutionnels et des mécanismes comme le « replay d'expérience », s'inspirent souvent des découvertes neuroscientifiques. Cette interconnexion est bénéfique dans le domaine du langage, où une compréhension plus profonde des capacités cognitives humaines uniques, telles que le traitement de structures linguistiques complexes, peut ouvrir la voie à des systèmes de traitement du langage plus sophistiqués. L'émergence de riches ensembles de données neuroimagerie naturalistes (par exemple, fMRI, MEG) aux côtés de modèles de langage avancés ouvre de nouvelles voies pour aligner les modèles de langage computationnels sur l'activité cérébrale humaine. Cependant, le défi réside dans le discernement des caractéristiques du modèle qui reflètent le mieux les processus de compréhension du langage dans le cerveau, soulignant ainsi l'importance d'intégrer des mécanismes inspirés de la biologie dans les modèles computationnels.En réponse à ce défi, la thèse introduit un cadre basé sur les données qui comble le fossé entre le traitement neurolinguistique observé dans le cerveau humain et les mécanismes computationnels des systèmes de traitement automatique du langage naturel (TALN). En établissant un lien direct entre les techniques d'imagerie avancées et les processus de TALN, elle conceptualise le traitement de l'information cérébrale comme une interaction dynamique de trois composantes critiques : le « quoi », le « où » et le « quand », offrant ainsi des perspectives sur la manière dont le cerveau interprète le langage lors de l'engagement avec des récits naturalistes. L'étude fournit des preuves convaincantes que l'amélioration de l'alignement entre l'activité cérébrale et les systèmes de TALN offre des avantages mutuels aux domaines de la neurolinguistique et du TALN. La recherche montre comment ces modèles computationnels peuvent émuler les capacités de traitement du langage naturel du cerveau en exploitant les technologies de réseau neuronal de pointe dans diverses modalités - langage, vision et parole. Plus précisément, la thèse met en lumière comment les modèles de langage pré-entraînés modernes parviennent à un alignement plus étroit avec le cerveau lors de la compréhension de récits. Elle examine le traitement différentiel du langage à travers les régions cérébrales, le timing des réponses (délais HRF) et l'équilibre entre le traitement de l'information syntaxique et sémantique. En outre, elle explore comment différentes caractéristiques linguistiques s'alignent avec les réponses cérébrales MEG au fil du temps et constate que cet alignement dépend de la quantité de contexte passé, indiquant que le cerveau code les mots légèrement en retard par rapport à celui actuel, en attendant plus de contexte futur. De plus, elle met en évidence la plausibilité biologique de l'apprentissage des états de réservoir dans les réseaux à état d'écho, offrant ainsi une interprétabilité, une généralisabilité et une efficacité computationnelle dans les modèles basés sur des séquences. En fin de compte, cette recherche apporte des contributions précieuses à la neurolinguistique, à la neuroscience cognitive et au TALN
This thesis explores the synergy between artificial intelligence (AI) and cognitive neuroscience to advance language processing capabilities. It builds on the insight that breakthroughs in AI, such as convolutional neural networks and mechanisms like experience replay 1, often draw inspiration from neuroscientific findings. This interconnection is beneficial in language, where a deeper comprehension of uniquely human cognitive abilities, such as processing complex linguistic structures, can pave the way for more sophisticated language processing systems. The emergence of rich naturalistic neuroimaging datasets (e.g., fMRI, MEG) alongside advanced language models opens new pathways for aligning computational language models with human brain activity. However, the challenge lies in discerning which model features best mirror the language comprehension processes in the brain, underscoring the importance of integrating biologically inspired mechanisms into computational models. In response to this challenge, the thesis introduces a data-driven framework bridging the gap between neurolinguistic processing observed in the human brain and the computational mechanisms of natural language processing (NLP) systems. By establishing a direct link between advanced imaging techniques and NLP processes, it conceptualizes brain information processing as a dynamic interplay of three critical components: "what," "where," and "when", offering insights into how the brain interprets language during engagement with naturalistic narratives. This study provides compelling evidence that enhancing the alignment between brain activity and NLP systems offers mutual benefits to the fields of neurolinguistics and NLP. The research showcases how these computational models can emulate the brain’s natural language processing capabilities by harnessing cutting-edge neural network technologies across various modalities—language, vision, and speech. Specifically, the thesis highlights how modern pretrained language models achieve closer brain alignment during narrative comprehension. It investigates the differential processing of language across brain regions, the timing of responses (Hemodynamic Response Function (HRF) delays), and the balance between syntactic and semantic information processing. Further, the exploration of how different linguistic features align with MEG brain responses over time and find that the alignment depends on the amount of past context, indicating that the brain encodes words slightly behind the current one, awaiting more future context. Furthermore, it highlights grounded language acquisition through noisy supervision and offers a biologically plausible architecture for investigating cross-situational learning, providing interpretability, generalizability, and computational efficiency in sequence-based models. Ultimately, this research contributes valuable insights into neurolinguistics, cognitive neuroscience, and NLP

Un modèle théorique du cortex cérébral est proposé. Il est basé sur les connaissances actuelles en neurobiologie et fournit un type original de réseau de neurones pour intégrer les différentes fonctions de l'intelligence artificielle : reconnaissance des formes, positionnement moteur dans l'espace, programmation et langage. Ce modèle peut aider à résoudre les problèmes de communication entre des traitements concurrents effectues sur des ensembles hétérogènes d'informations, usuelles, auditives, motrices et symboliques.

碩士
國立臺南大學
數位學習科技學系碩士班
98
In this study, the research theme is event-related potentials (Event-Related Potentials, ERPs) for the spoken Chinese word. When people experience the events of daily life, the memory store in the brain.The brain''s memory process and extraction is the subject of this study. The experimental procedure is divided into two steps, the first step is oddball test. The main purpose of oddball test is to check the state of the brain’s reaction, if the test results correctly then enter the next step. The second step is spoken Chinese cognitive experiments, the user watches a randomly selected film, and the content fo the film is the target. There are three different stimuli, of which only one related to the selected film content. The ERPs results show that, Chinese spoken language evoke N200 and P300 components, known as the N2-P3 complex. The classification rules are established by The algorithm of CART. When the N2-P3 complex amplitude at Cz electrode is maximum, the event will be determined the target, and the success rate is 81%.

In unserem alltäglichen Leben ist Sprache ein unerlässliches Mittel für Kommunikation und die Umsetzung sozialer Interaktionen. Sprache kann in zwei verschiedene Modalitäten unterteilt werden, in die auditorische und die visuelle Modalität. Die auditorische Modalität umfasst gesprochene Sprache, wohingegen die visuelle Modalität vom geschriebenen Teil der Sprache gebildet wird. Auch wenn ein Tag ohne Sprechen für die meisten von uns unvorstellbar ist, hat die bisherige Forschung die Untersuchung von Effekten bei der Verarbeitung von emotionalem Bedeutungsinhalt in gesprochener Sprache, im Gegensatz zu der Verarbeitung von geschriebener Sprache, vernachlässigt. Die Verarbeitung des emotionalen Bedeutungsinhalts von geschriebenen Wörtern hat eine Vielzahl von Studien mit Hilfe von ereigniskorrelierten Potentialen (EKPs) ausführlich untersucht. Im Gegensatz dazu wurde der emotionale Bedeutungsinhalt bei der Verarbeitung von gesprochener Sprache nur gelegentlich und meist entweder in seiner Interaktion mit emotionaler Prosodie oder fokussiert auf die Existenz einer spezifischen EKP Komponente untersucht. Daher bleibt die Frage offen, wie und an welchen Verarbeitungsschritten der emotionale Inhalt gesprochener Sprache ereigniskorrelierte Potentiale beeinflusst, unabhängig von emotionaler Prosodie und der Frage, ob Gemeinsamkeiten mit der Verarbeitung von geschriebenen emotionalen Wörtern bestehen. In dieser Dissertation untersuche ich die Verarbeitung von gesprochenen Einzelwörtern mit positivem, neutralem und negativem Inhalt, mit der erkenntnisleitenden Fragestellung, ob der emotionale Inhalt von gesprochenen Wörtern Emotionseffekte in EKPs hervorruft und ob diese vergleichbar sind zu denen, die für geschriebene Wörter gezeigt wurden. In der ersten dieser Dissertation zugrundeliegenden Studie wurden gesprochene Wörter mit emotionalem und neutralem Inhalt den Versuchspersonen in zwei verschiedenen Lautstärken präsentiert, um mögliche Interaktionen mit bottom-up Aufmerksamkeitseffekten, geleitet durch die Größe des Stimulus, zu erklären. Für visuelle Stimuli mit emotionalem Inhalt, wie Bilder oder geschriebene Wörter, hat die Größe des Stimulus erhöhte emotions-bedingte EKPs hervorgerufen, zum Beispiel auf der Ebene der early posterior negativity (EPN). Es wurde untersucht, ob diese erhöhte Relevanz von größeren visuellen Stimuli auf die auditorische Modalität übertragbar sein könnte. Negativer emotionaler Bedeutungsinhalt führt zu einer erhöhten frontalen Positivierung und einer parieto-okzipitalen Negativierung zwischen 370 und 530 Millisekunden. Diese Komponente zeigt Ähnlichkeit mit der visuellen EPN, obwohl sich die Negativierung zu zentraleren Arealen der Kopfoberfläche ausweitet. Daher stellt sich die Frage, ob diese Komponente das auditorische Pendant zu einer visuellen EPN darstellen könnte. Entscheidend ist hier, dass keine Interaktion dieser emotions-bedingten EKP Komponente mit dem Lautstärkefaktor beobachtet werden kann. Die folgenden Vergleichsaspekte deuten auf umfassendere Unterschiede zwischen visueller und auditorischer Sprachverarbeitung hin: die fehlende Interaktion zwischen der Größe des Stimulus und der Emotionseffekte, die Unterschiede in den Topographien der Emotionseffekte sowie unterschiedliche Latenzen verglichen zu der visuellen EPN. Der zweite Teil dieser Dissertation ist auf einen direkteren Vergleich von Emotionseffekten in der visuellen und auditorischen Modalität ausgerichtet. Zu diesem Zweck wurde eine zweite Studie durchgeführt, in der Versuchspersonen dieselben Wörter in geschriebener und gesprochener Modalität präsentiert bekamen. Die gesprochenen Wörter wurden dabei sowohl von einer Computerstimme (Experiment 1) als auch von einer menschlichen Stimme (Experiment 2) produziert. Diese Studie wurde konzipiert, um die Existenz einer „auditorischen EPN“ und ihre Randbedingungen zu untersuchen. Darüber hinaus sollte die These überprüft werden, ob die höhere soziale Relevanz einer menschlichen Stimme die Emotionseffekte vergrößert. In beiden Experimenten zeigen sich Emotionseffekte. Für geschriebene Wörter zwischen 230 und 400 Millisekunden, im Zeitbereich der early posterior negativity, für gesprochene Wörter zwischen 460 und 510 Millisekunden. Wenn man die Verteilung der EKP Differenzen zwischen emotionalen und neutralen auditorischen Wörtern berücksichtigt, zeigen die Effekte interessanterweise sogar eine größere Ähnlichkeit mit der visuellen EPN als die Ergebnisse des ersten Teils dieser Dissertation. Eine Quellenlokalisierung ergab vergleichbare neuronale Generatoren im superioren parietalen Lobus (SPL) und im inferioren temporalen Lobus (IPL), sowohl im visuellen als auch im „auditorischen EPN“ Zeitfenster. Diese Befunde deuten auf Gemeinsamkeiten in der Verarbeitung emotionaler Inhalte über die Modalitäten hinweg hin, die – zumindest teilweise – durch das gleiche neuronale System gestützt werden. Trotzdem erscheinen diese Gemeinsamkeiten überraschend, da für die visuelle EPN angenommen wird, dass sie eine verstärkte sensorische Enkodierung für emotionale Stimuli in visuellen Arealen abbildet. Die oben beschriebenen und in diesen Studien gezeigten Emotionseffekte unterscheiden sich bezüglich ihrer Latenzen, Topographien und der Valenz, welche den Effekt hervorruft (positiv oder negativ). Im letzten Teil der Dissertation wurden daher systematisch Unterschiede zwischen den Studien untersucht um potenzielle Ursachen für die oben aufgeführten Unterschiede in den Emotionseffekten bestimmen zu können. Es zeigen sich Geschlechterunterschiede in den Topographien in Studie 2, die jedoch nicht die gefundenen Unterscheide in den Emotionseffekten zwischen den beiden Studien erklären können. Es wird angenommen, dass beide Studien die gleiche auditorische emotions-bedingte Komponente (AEK) in einem vergleichbaren Zeitfenster (Studie 1: 477 530 ms; Studie 2: 464 515 ms) hervorrufen, welcher in der ersten Studie eine N400-ähnlichen Verteilung vorausgegangen ist. Obwohl keine Interaktionen zwischen emotionalem Inhalt und Lautstärke aufgezeigt werden können, gehe ich davon aus, dass die Manipulation der Lautstärke in der ersten Studie den Kontext des Experiments verändert, und so den früheren Effekt ausgelöst hat. Auch wenn keine verifizierbaren Ursachen für die beschriebenen Unterschiede zwischen den Emotionseffekten aufgezeigt werden konnten, ist es mir mit dieser Dissertation gelungen, die Existenz einer auditorischen emotions-bedingten Komponente zu zeigen, die durch emotionalen (in Vergleich zu neutralem) Inhalt während der Verarbeitung von gesprochener Sprache hervorgerufen wird. Diese Komponente spiegelt sich in einer anterioren Positivierung und einer posterioren Negativierung zwischen 460 und 520 Millisekunden nach Wortbeginn wider. Diese zeigt sich gleichbleibend, unabhängig von der sozialen Signifikanz der Stimme des Sprechers oder der Manipulation der Lautstärke. Bezüglich eines Vergleich des zugrundeliegenden neuronalen Netzwerkes während der Verarbeitung des Inhalts von gesprochenen und geschriebenen Wörtern, kann man annehmen, dass die Verarbeitung Hirnareale aktiviert, die zumindest teilweise im SPL und IPL liegen. Obwohl die Verteilung der AEK eine hohe Ähnlichkeit zur visuellen EPN aufzeigt, kann man nicht annehmen, dass dieser Effekt ein auditorisches Pendant darstellt. Diese Schlussfolgerung beruht darauf, dass sich eine typische EPN-Verteilung nur bei der Berechnung der Differenzkurven von emotionalen und neutralen Stimuli zeigt. Die daraus resultierende posteriore Negativierung spiegelt eine erhöhte Aktivierung von visuellen Arealen - hervorgerufen durch emotionale Stimuli - wider. Die Analyse der zugrundeliegenden neuronalen Generatoren für den Unterschied zwischen auditorischen emotionalen und neutralen Stimuli liefert keine signifikanten Ergebnisse. Trotzdem zeigen die zugrundeliegenden Topographien der einzelnen Emotionskategorien, dass die Gemeinsamkeit auf der Ebene der Differenzkurven aus völlig unterschiedlichen Verteilungen resultiert. Zukünftige Forschung müsste das auditorische Stimulusmaterial bezüglich der Wortlänge oder des Worterkennungspunktes strikter kontrollieren, um den zeitlichen Jitter in den Daten zu reduzieren und somit die neuronalen Generatoren einer auditorischen emotions-bedingten Komponente besser bestimmen zu können.

Deep Neural Networks (DNN) inspired by the human brain have redefined the state-of-the-art performance in AI during the past decade. Much of the research is still trying to understand and explain the function of these networks. In this thesis, we leverage knowledge from the neuroscience literature to evaluate the representations learned in state-of-the-art language models. We use sentences with simple syntax and semantics (e.g., “The bone was eaten by the dog.”), and train multiple neural networks to predict the part of speech, next word. We present other sentences of this same simple form, word-by-word to humans in a magnetoencephalography (MEG) scanner for silent reading and comprehension. We then train a linear regression model to predict observed brain recording from the hidden layers of the trained neural networks and popular pre-trained networks like BERT and ELMo. We find that the middle layers of these networks are the most predictive of the recorded brain activity. But, a more fine-grained evaluation shows that various types of stimuli (determiner, adjective, noun, verb) are represented more dominantly in different layers of the language model. Further, we test the semantic composition capabilities of these networks with respect to the human brain. Semantic composition is defined as the rule-based combination of the parts that constitutes the meaning of the whole. We collect new data and develop a new framework to perform this evaluation incrementally as each word in the sentence is processed in the brain and DNN. As a result, we are able to analyze the effect of the composition function in representing the same word as more of the sentence context becomes available. Our experiments show that DNN models are effective in encoding the sentence being read and are able to predict the word which occurred earlier in the sentence, indicating good composition. We find that in these tests, the right frontal and right temporal brain regions are predicted with best accuracy. Previous research has suggested that these brain regions are responsible for executive and memory function. As an additional contribution, we propose a new dynamic time warping based distance metric to evaluate alignment between the predicted brain activity versus the observed brain activity. The new metric helps tackle the variability observed in a single subject’s recorded brain activity.
Ministry of Human Resource and development India (MHRD), Pratiksha Trust, and CMU BrainHub

Dementia is a growing challenge to our society. Research suggests a number of modifiable factors are associated with the risk of developing this condition. One such modifiable factor is physical activity. Physical activity has been associated with both brain structure and cognition. Further evidence suggests an association between brain structure and cognition. While the majority of neuroimaging studies have used volumetric MRI measures to examine brain structure, an emerging alternative is to examine cortical sulcal characteristics. This study sought to determine how sulcal characteristics relate to physical activity and cognition. A final sample of 320 participants aged between 64 and 70 years were selected from an observational study of lifestyle factors including MRI and cognitive data. The results presented here indicate that physical activity predicts differences in sulcal structure. Width of the left superior frontal sulcus, negatively correlated with physical activity, was associated with improved processing speed and executive function. These findings are consistent with the literature showing that physical activity is beneficial in preventing against cognitive decline and provides important information about the usefulness of sulcal characteristics in the investigation of cerebral and cognitive health.