Thèses sur le sujet « Multimodal Concepts »

En recherche d'images aujourd'hui, nous manipulons souvent de grands volumes d'images, qui peuvent varier ou même arriver en continu. Dans une base d'images, on se retrouve ainsi avec certaines images anciennes et d'autres nouvelles, les premières déjà indexées et possiblement annotées et les secondes en attente d'indexation ou d'annotation. Comme la base n'est pas annotée uniformément, cela rend l'accès difficile par le biais de requêtes textuelles. Nous présentons dans ce travail différentes techniques pour interagir, naviguer et rechercher dans ce type de bases d'images. Premièrement, un modèle d'interaction à court terme est utilisé pour améliorer la précision du système. Deuxièmement, en se basant sur un modèle d'interaction à long terme, nous proposons d'associer mots textuels et caractéristiques visuelles pour la recherche d'images par le texte, par le contenu visuel, ou mixte texte/visuel. Ce modèle de recherche d'images permet de raffiner itérativement l'annotation et la connaissance des images. Nous identifions quatre contributions dans ce travail. La première contribution est un système de recherche multimodale d'images qui intègre différentes sources de données, comme le contenu de l'image et le texte. Ce système permet l'interrogation par l'image, l'interrogation par mot-clé ou encore l'utilisation de requêtes hybrides. La deuxième contribution est une nouvelle technique pour le retour de pertinence combinant deux techniques classiques utilisées largement dans la recherche d'information~: le mouvement du point de requête et l'extension de requêtes. En profitant des images non pertinentes et des avantages de ces deux techniques classiques, notre méthode donne de très bons résultats pour une recherche interactive d'images efficace. La troisième contribution est un modèle nommé "Sacs de KVR" (Keyword Visual Representation) créant des liens entre des concepts sémantiques et des représentations visuelles, en appui sur le modèle de Sac de Mots. Grâce à une stratégie d'apprentissage incrémental, ce modèle fournit l'association entre concepts sémantiques et caractéristiques visuelles, ce qui contribue à améliorer la précision de l'annotation sur l'image et la performance de recherche. La quatrième contribution est un mécanisme de construction incrémentale des connaissances à partir de zéro. Nous ne séparons pas les phases d'annotation et de recherche, et l'utilisateur peut ainsi faire des requêtes dès la mise en route du système, tout en laissant le système apprendre au fur et à mesure de son utilisation. Les contributions ci-dessus sont complétées par une interface permettant la visualisation et l'interrogation mixte textuelle/visuelle. Même si pour l'instant deux types d'informations seulement sont utilisées, soit le texte et le contenu visuel, la généricité du modèle proposé permet son extension vers d'autres types d'informations externes à l'image, comme la localisation (GPS) et le temps.

Cette thèse considère l'apprentissage de motifs récurrents dans la perception multimodale. Elle s'attache à développer des modèles robotiques de ces facultés telles qu'observées chez l'enfant, et elle s'inscrit en cela dans le domaine de la robotique développementale.Elle s'articule plus précisément autour de deux thèmes principaux qui sont d'une part la capacité d'enfants ou de robots à imiter et à comprendre le comportement d'humains, et d'autre part l'acquisition du langage. A leur intersection, nous examinons la question de la découverte par un agent en développement d'un répertoire de motifs primitifs dans son flux perceptuel. Nous spécifions ce problème et établissons son lien avec ceux de l'indétermination de la traduction décrit par Quine et de la séparation aveugle de source tels qu'étudiés en acoustique.Nous en étudions successivement quatre sous-problèmes et formulons une définition expérimentale de chacun. Des modèles d'agents résolvant ces problèmes sont également décrits et testés. Ils s'appuient particulièrement sur des techniques dites de sacs de mots, de factorisation de matrices et d'apprentissage par renforcement inverse. Nous approfondissons séparément les trois problèmes de l'apprentissage de sons élémentaires tels les phonèmes ou les mots, de mouvements basiques de danse et d'objectifs primaires composant des tâches motrices complexes. Pour finir nous étudions le problème de l'apprentissage d'éléments primitifs multimodaux, ce qui revient à résoudre simultanément plusieurs des problèmes précédents. Nous expliquons notamment en quoi cela fournit un modèle de l'ancrage de mots acoustiques
This thesis focuses on learning recurring patterns in multimodal perception. For that purpose it develops cognitive systems that model the mechanisms providing such capabilities to infants; a methodology that fits into thefield of developmental robotics.More precisely, this thesis revolves around two main topics that are, on the one hand the ability of infants or robots to imitate and understand human behaviors, and on the other the acquisition of language. At the crossing of these topics, we study the question of the how a developmental cognitive agent can discover a dictionary of primitive patterns from its multimodal perceptual flow. We specify this problem and formulate its links with Quine's indetermination of translation and blind source separation, as studied in acoustics.We sequentially study four sub-problems and provide an experimental formulation of each of them. We then describe and test computational models of agents solving these problems. They are particularly based on bag-of-words techniques, matrix factorization algorithms, and inverse reinforcement learning approaches. We first go in depth into the three separate problems of learning primitive sounds, such as phonemes or words, learning primitive dance motions, and learning primitive objective that compose complex tasks. Finally we study the problem of learning multimodal primitive patterns, which corresponds to solve simultaneously several of the aforementioned problems. We also details how the last problems models acoustic words grounding

Memoria (licenciado en ciencias jurídicas y sociales)
El éxito de las transacciones comerciales nacionales e internacionales depende, entre muchos otros factores, de la eficiencia de las cadenas de transporte de las mercaderías transadas. En la actualidad el transporte ha tenido que adaptarse a las exigencias de la “nueva economía”, cuya demanda por la integración económica, permiten la creación de la figura jurídica del transporte multimodal de mercancías. El transporte multimodal de mercancías es aquel por el cual se transportan mercancías, por dos o más modos diferentes de transporte, de acuerdo a un sólo contrato. La presente obra tiene como objetivo presentar e introducir el novedoso concepto del transporte multimodal, diferenciándolo del tradicional transporte unimodal y segmentado; plantear sus actuales dificultades y divergencias, como también proponer los cambios necesarios para la implementación del mismo como nuevo sistema de transporte que aporte al crecimiento económico de los países en vías de desarrollo.

Cette these s'inscrit dans le domaine de la communication homme-machine et plus specifiquement dans celui de la conception et de la realisation des interfaces multimodales. Les travaux de recherche presentes decrivent les nouveaux problemes poses par ce type d'interfaces et proposent des solutions qui sont testees a travers des realisations abouties. Le travail est divise en trois etapes. La premiere, a caractere exploratoire, decrit la conception et la realisation d'une interface multimodale pour une application de dessin graphique (limsi-draw). Cette premiere experimentation permet de reveler des problemes importants lies aux contraintes technologiques actuelles, de souligner l'importance du facteur temporel souvent neglige dans les interfaces classiques et de proposer une methode efficace pour la fusion des informations. Le modele d'architecture adopte est articule autour d'interpreteurs independants et d'un controleur de dialogue central utilisant des regles de decision qui permettent d'assurer une fusion robuste. Cette partie de l'etude se termine par une evaluation avec des sujets humains d'ou sont degages des enseignements interessants sur l'utilisation des modalites. La seconde etape a pour but la conception d'un outil pour la specification des interactions multimodales. Cet outil, baptise specimen, est fonde sur un modele combinant une specification par des reseaux de transitions augmentes a une specification par messages a l'aide d'operateurs de composition permettant de decrire des actions sequentielles et/ou paralleles. Par ailleurs, l'elaboration d'une methode de detection de messages repartie a travers des agents specialises permet la definition de mecanismes de fusion generaux. Dans la derniere etape, specimen est applique pour la construction d'une interface multimodale pour non-voyants (meditor). L'objectif vise est double: d'une part valider cet outil a travers une realisation concrete, d'autre part etudier l'apport de la multimodalite au probleme de l'acces aux systemes informatiques par des utilisateurs non-voyants. Des resultats preliminaires encourageants sont obtenus et des perspectives prometteuses pour une communication homme-machine intelligente combinant des modeles d'interaction anthropomorphiques et physiques sont discutees en conclusion

This paper presents a study of educational materials used in Swedish language education. The aim of the study is to understand in which way multimodal resources can be used in texts, to benefit in the process of meaning making among pupils in the nine-year compulsory school. The theoretical framework used as a basis for understanding and analysing these educational materials is the social semiotic multimodal perspective and the design theoretical multimodal perspective. The study is a multimodal text analysis but it also involves analyses of the syllabi connected to the subject of Swedish language education. The extended concept of text was introduced in the year of 2000 in the syllabus and today multimodal texts are supposed to be part of the Swedish language education. In course of this study the researcher found that multimodal resources can be used in different ways to benefit in the process of meaning making. The study shows that the use of resources is connected to the different aims among texts and to the affordances of meaning making resources.  The aim of texts differs among and in-between the educational materials connected to the different syllabi.  The researcher also found that the texts supposed to be included in Swedish language education has increased since the extended concept of text was introduced and according to the process of time. Pupils however, aren´t introduced to strategies for dealing with these new kinds of texts, in the same extent.

Cette thèse de doctorat est consacrée à la reconnaissance de concepts visuels (VCR pour "Visual Concept Recognition"). En raison des nombreuses difficultés qui la caractérisent, cette tâche est toujours considérée comme l’une des plus difficiles en vision par ordinateur et reconnaissance de formes. Dans ce contexte, nous avons proposé plusieurs contributions, particulièrement dans le cadre d’une approche de reconnaissance multimodale combinant efficacement les informations visuelles et textuelles. Tout d’abord, nous avons étudié différents types de descripteurs visuels de bas-niveau sémantique pour la tâche de VCR incluant des descripteurs de couleur, de texture et de forme. Plus précisément, nous pensons que chaque concept nécessite différents descripteurs pour le caractériser efficacement pour permettre sa reconnaissance automatique. Ainsi, nous avons évalué l’efficacité de diverses représentations visuelles, non seulement globales comme la couleur, la texture et la forme, mais également locales telles que SIFT, Color SIFT, HOG, DAISY, LBP et Color LBP. Afin de faciliter le franchissement du fossé sémantique entre les descripteurs bas-niveau et les concepts de haut niveau sémantique, et particulièrement ceux relatifs aux émotions, nous avons proposé des descripteurs visuels de niveau intermédiaire basés sur l’harmonie visuelle et le dynamisme exprimés dans les images. De plus, nous avons utilisé une décomposition spatiale pyramidale des images pour capturer l’information locale et spatiale lors de la construction des descripteurs d’harmonie et de dynamisme. Par ailleurs, nous avons également proposé une nouvelle représentation reposant sur les histogrammes de couleur HSV en utilisant un modèle d’attention visuelle pour identifier les régions d’intérêt dans les images. Ensuite, nous avons proposé un nouveau descripteur textuel dédié au problème de VCR. En effet, la plupart des photos publiées sur des sites de partage en ligne (Flickr, Facebook, ...) sont accompagnées d’une description textuelle sous la forme de mots-clés ou de légende. Ces descriptions constituent une riche source d’information sur la sémantique contenue dans les images et il semble donc particulièrement intéressant de les considérer dans un système de VCR. Ainsi, nous avons élaboré des descripteurs HTC ("Histograms of Textual Concepts") pour capturer les liens sémantiques entre les concepts. L’idée générale derrière HTC est de représenter un document textuel comme un histogramme de concepts textuels selon un dictionnaire (ou vocabulaire), pour lequel chaque valeur associée à un concept est l’accumulation de la contribution de chaque mot du texte pour ce concept, en fonction d’une mesure de distance sémantique. Plusieurs variantes de HTC ont été proposées qui se sont révélées être très efficaces pour la tâche de VCR. Inspirés par la démarche de l’analyse cepstrale de la parole, nous avons également développé Cepstral HTC pour capturer à la fois l’information de fréquence d’occurrence des mots (comme TF-IDF) et les liens sémantiques entre concepts fournis par HTC à partir des mots-clés associés aux images. Enfin, nous avons élaboré une méthode de fusion (SWLF pour "Selective Weighted Later Fusion") afin de combiner efficacement différentes sources d’information pour le problème de VCR. Cette approche de fusion est conçue pour sélectionner les meilleurs descripteurs et pondérer leur contribution pour chaque concept à reconnaître. SWLF s’est révélé être particulièrement efficace pour fusion des modalités visuelles et textuelles, par rapport à des schémas de fusion standards. [...]
This Ph.D thesis is dedicated to visual concept recognition (VCR). Due to many realistic difficulties, it is still considered to be one of the most challenging problems in computer vision and pattern recognition. In this context, we have proposed some innovative contributions for the task of VCR, particularly in building multimodal approaches that efficiently combine visual and textual information. Firstly, we have proposed semantic features for VCR and have investigated the efficiency of different types of low-level visual features for VCR including color, texture and shape. Specifically, we believe that different concepts require different features to efficiently characterize them for the recognition. Therefore, we have investigated in the context of VCR various visual representations, not only global features including color, shape and texture, but also the state-of-the-art local visual descriptors such as SIFT, Color SIFT, HOG, DAISY, LBP, Color LBP. To help bridging the semantic gap between low-level visual features and high level semantic concepts, and particularly those related to emotions and feelings, we have proposed mid-level visual features based on the visual harmony and dynamism semantics using Itten’s color theory and psychological interpretations. Moreover, we have employed a spatial pyramid strategy to capture the spatial information when building our mid-level features harmony and dynamism. We have also proposed a new representation of color HSV histograms by employing a visual attention model to identify the regions of interest in images. Secondly, we have proposed a novel textual feature designed for VCR. Indeed, most of online-shared photos provide textual descriptions in the form of tags or legends. In fact, these textual descriptions are a rich source of semantic information on visual data that is interesting to consider for the purpose of VCR or multimedia information retrieval. We propose the Histograms of Textual Concepts (HTC) to capture the semantic relatedness of concepts. The general idea behind HTC is to represent a text document as a histogram of textual concepts towards a vocabulary or dictionary, whereas its value is the accumulation of the contribution of each word within the text document toward the underlying concept according to a predefined semantic similarity measure. Several variants of HTC have been proposed that revealed to be very efficient for VCR. Inspired by the Cepstral speech analysis process, we have also developed Cepstral HTC to capture both term frequency-based information (like TF-IDF) and the relatedness of semantic concepts in the sparse image tags, which overcomes the HTC’s shortcoming of ignoring term frequency-based information. Thirdly, we have proposed a fusion scheme to combine different sources of Later Fusion, (SWLF) is designed to select the best features and to weight their scores for each concept to be recognized. SWLF proves particularly efficient for fusing visual and textual modalities in comparison with some other standard fusion schemes. While a late fusion at score level is reputed as a simple and effective way to fuse features of different nature for machine-learning problems, the proposed SWLF builds on two simple insights. First, the score delivered by a feature type should be weighted by its intrinsic quality for the classification problem at hand. Second, in a multi-label scenario where several visual concepts may be assigned to an image, different visual concepts may require different features which best recognize them. In addition to SWLF, we also propose a novel combination approach based on Dempster-Shafer’s evidence theory, whose interesting properties allow fusing different ambiguous sources of information for visual affective recognition. [...]

This dissertation presents new concepts in front end design for receivers with large, multiband tuning ranges. Such receivers are required to support large bandwidths (up to 10's of MHz) over very large tuning ranges (30:1 and beyond) with antennas that are usually narrowband, or which at best support multiple narrow bandwidths. Traditional techniques to integrate a single antenna with such receivers are limited in their ability to handle simultaneous channels distributed over very large tuning ranges, which is important for frequency-agile cognitive radio, surveillance, and other applications requiring wideband or multiband monitoring. Direct conversion architecture is gaining popularity due to the recent advancements in CMOS--based RFIC technology. The possibility of multiple parallel transceivers in RF CMOS suggests an approach to antenna--receiver integration using multiplexers. This dissertation describes an improved use of multiplexers to integrate antennas to receivers. First, the notion of sensitivity--constrained design is considered. In this approach, the goal is first to achieve sensitivity which is nominally dominated by external (environmental) noise, and then secondly to improve bandwidth to the maximum possible consistent with this goal. Next, a procedure is developed for designing antenna-multiplexer-preamplifier assemblies using this philosophy. It is shown that the approach can significantly increase the usable bandwidth and number of bands that can be supported by a single, traditional antenna. This performance is verified through field experiments. A prototype multiband multimode radio for public safety applications using these concepts is designed and demonstrated.
Ph. D.

Thesis (Ph. D.)--Electrical and Computer Engineering, Georgia Institute of Technology, 2005.
Ralph, Stephen E., Committee Chair ; McLaughlin, Steven W., Committee Member ; Buck, John A., Committee Member ; Laskar, Joy, Committee Member ; Raman, Chandra, Committee Member. Vita. Includes bibliographical references.

Submitted by Mario BC (mario@bc.ufrpe.br) on 2018-08-14T14:12:11Z No. of bitstreams: 1 Tereza Cristina Cavalcanti de Albuquerque.pdf: 8612345 bytes, checksum: 63cefd3234e79440c4d81f8c47e647fe (MD5)
Made available in DSpace on 2018-08-14T14:12:11Z (GMT). No. of bitstreams: 1 Tereza Cristina Cavalcanti de Albuquerque.pdf: 8612345 bytes, checksum: 63cefd3234e79440c4d81f8c47e647fe (MD5) Previous issue date: 2018-02-05
In today's language, several semiotic modes (verbal, visual, sonorous and gestural, among others) are integrated to construct a meaning, thus composing a multimodal text that together with advances in technology occupy more spaces each day, and among them, the space of teaching and learning of sciences. However, although there is a growing use of multimodal text, it is still not being exploited to its full potential, as teachers are often not instrumental in reading to understand and produce. Above all, in considering the eminently abstract and symbolic character of its contents, the initial formation of science teachers demands the reading and production of more contextualized multimodal texts that can generate effective and qualified actions in the teaching practice regarding the use of different modes of language. In this context, the objective of this study was to analyze how the knowledge about Multimodality inserted in the initial formation of teachers can contribute to the reading and the construction of multimodal texts for teaching about human lung breathing. In the course of this research the elective Multimodality in Science Education course was developed, offered for the undergraduate courses in Chemistry and Biological Sciences, of a Brazilian public university. The data were constructed during 11 months of work incorporated to this discipline, with a group composed by four students. The methodology that guided the present study was an adaptation of the Collaborative Research (IBIAPINA, 2008) and the data analysis employed the categories of Visual Design Grammar (KRESS and van LEEUWEN, 2006). The productions of the analyzed students were: multimodal texts produced for videotapes on pulmonary respiration; answers to the interview about the process of videotape production; video-recorded analysis of the multimodal texts related to the respiratory system and hematose used in videotapes; production of the Set of Multimodal Texts for Teaching Human Pulmonary Respiration (CTMER) and validation of CTMER in class at graduation. At the same time, the researcher analyzed the multimodal texts on lung breathing in the textbooks. The results indicate that the implementation of Multimodality knowledge in reading, evaluation and production of multimodal texts increased the capacity of representation and communication of the participants involved through the integrated use of different modes of language and enabled the construction of didactic material (CTMER) to teach the concept of human lung respiration integrated with the environment and other organic systems.
Na linguagem atual, vários modos semióticos (verbal, visual, sonoro e gestual, entre outros) se integram para a construção de um significado, compondo assim, um texto multimodal que em conjunto com os avanços da tecnologia ocupam mais espaços a cada dia, e dentre eles, o espaço de ensino e aprendizagem de ciências. No entanto, embora exista um crescente uso do texto multimodal, este ainda não está sendo explorado em seu pleno potencial, pois muitas vezes os professores não estão instrumentalizados para a leitura necessária à sua compreensão e à sua produção. Sobretudo, ao se considerar o caráter eminentemente abstrato e simbólico de seu conteúdo, a formação inicial de professores de ciências demanda a leitura e a produção de textos multimodais mais contextualizados que possa gerar ações efetivas e qualificadas na prática docente referente ao uso dos diferentes modos de linguagem. Neste contexto, o objetivo deste estudo foi analisar como os conhecimentos sobre a Multimodalidade inseridos na formação inicial de professores podem contribuir para a leitura e a construção de textos multimodais para o ensino sobre respiração pulmonar humana. No percurso desta investigação foi desenvolvida a disciplina eletiva Multimodalidade na Educação em Ciências, ofertada para os cursos de Licenciatura em Química e em Ciências Biológicas, de uma universidade pública brasileira. Os dados foram construídos ao longo de 11 meses de trabalho incorporados a esta disciplina, com um grupo composto por quatro estudantes. A metodologia que orientou o presente estudo foi uma adaptação da Pesquisa Colaborativa (IBIAPINA, 2008) e a análise dos dados empregou as categorias da Gramática do Design Visual (KRESS e van LEEUWEN, 2006). As produções dos estudantes analisadas foram: textos multimodais produzidos para videoaulas sobre respiração pulmonar; respostas à entrevista sobre o processo de produção das videoaulas; análise vídeo-gravada sobre os textos multimodais relativos ao sistema respiratório e hematose empregados nas videoaulas; produção do Conjunto de Textos Multimodais para o Ensino sobre Respiração Pulmonar Humana (CTMER) e validação do CTMER em aula na graduação. Ao mesmo tempo, foram realizadas análises dos textos multimodais sobre respiração pulmonar presentes nos livros didáticos, pela pesquisadora. Os resultados indicam que a implementação dos conhecimentos da Multimodalidade na leitura, na avaliação e na produção de textos multimodais ampliou a capacidade de representação e comunicação dos participantes envolvidos através do uso integrado de diferentes modos de linguagem e possibilitou a construção de material didático (CTMER) para o ensino do conceito de respiração pulmonar humana integrado ao ambiente e aos demais sistemas orgânicos.

Acompanha: Unidade didática: entre o mais e o menos
Este trabalho analisa a aprendizagem significativa dos conceitos elétricos de corrente, resistência, tensão e potência elétrica a partir da Abordagem POE - Predizer, Observar e Explicar (Tao e Gustone, 1999; Oliveira, 2003), Aprendizagem Significativa (Ausubel, 2003; Moreira, 1999) e a Teoria de Multimodos e Múltiplas Representações (Prain; Waldrip, 2006; Laburú et al, 2011). Assim, o objetivo foi a utilização destes fundamentos teóricos como inspiração para a criação e aplicação da sequência didática durante as aulas da Disciplina de Princípios de Eletricidade e Magnetismo. Sendo a sequência didática o produto desta pesquisa, buscou-se formular aquela a fim de propor aos alunos diversas formas de representação de um mesmo conceito, inclusive com o uso de simulador. Para tanto, empregou-se o aplicativo de simulação Kit de Circuito DC, desenvolvido pelo projeto Tecnologia no Ensino de Física (PhET), da Universidade do Colorado. O corpus desta pesquisa restringiu-se a 14 dos 28 alunos, das turmas do 3º e 4º semestre de Engenharia de Produção e 4º semestre de Engenharia Mecânica, de uma instituição de ensino particular. Para a viabilização desta pesquisa optou-se pelo tipo de pesquisa exploratória. Os dados coletados foram avaliados segundo as metodologias quantitativa e qualitativa. Como resultados, os alunos demonstraram curiosidade no uso do painel elétrico e durante o uso do simulador. Observou-se que os alunos obtiveram um ganho conceitual superior ao ensino tradicional e equivalente ao ensino interativo, tanto para a aprendizagem demonstrada pelo questionário de conhecimentos básicos e pelo teste de concepções científicas sobre corrente elétrica (Silveira et al, 1989). No entanto, no discurso teórico sobre os conceitos básicos, mesmos após as orientações, muitos alunos mantiveram em sua estrutura cognitiva a interpretação do circuito relacionada às concepções alternativas.
This work analyzes the significant learning of electrical concepts of current, resistance, voltage and electric power from the POE Approach - Predicting, Observing and Explaining (Tao and Gustone, 1999; Oliveira, 2003), Significant Learning (Ausubel, 1999) and Multimode Theory and Multiple Representations Theory (Prain, Waldrip, 2006, Laburú et al, 2011). Thus, the objective was to use these theoretical foundations as inspiration for the creation and application of the didactic sequence during the classes of the Discipline of Principles of Electricity and Magnetism. Being the didactic sequence the product of this research, we tried to formulate that one in order to propose to the students several forms of representation of the same concept, even with the use of simulator. In order to do so, the DC Circuit Kit simulation application was developed, developed by the Technology in Teaching of Physics (PhET), University of Colorado. The corpus of this research was restricted to 14 of the 28 students, from the classes of the 3rd and 4th semester of Production Engineering and the 4th semester of Mechanical Engineering, from a private educational institution. In order to make this research viable, we opted for the type of exploratory research. The data collected were evaluated according to the quantitative and qualitative methodologies. As a result, students were curious about the use of the electrical panel and during the use of the simulator. It was observed that the students obtained a conceptual gain superior to the traditional teaching and equivalent to the interactive teaching, both for the learning demonstrated by the basic knowledge questionnaire and the test of scientific conceptions about electric current (Silveira et al, 1989). However, in the theoretical discourse on the basic concepts, same after the orientations, many students maintained in their cognitive structure the interpretation of the circuit related to the alternative conceptions.

Since the advent of the Internet, we have been observing a fast-paced development within the computing world. One of the major innovations in recent years is the “Internet of Things”, which brings interconnectedness between devices and humans to unprecedented heights. This technological breakthrough enabled the emergence of a new sub-field within Learning Analytics, Multimodal Learning Analytics, which makes use of several types of data sources to study learning-related processes. As computers and sensors become increasingly cheaper and more accessible,  research within this new sub-field grows, yet some gaps remain unexplored. Additionally, there is a research bias toward computer-assisted learning environments, rather than physical ones. At the same time, the current labor market is highly competitive, and possessing profession-related skills is not sufficient to land a job. Besides these skills, there is an increasing demand for social skills, such as communication, teamwork, and collaboration. However, there is a gap between the skills that are trained in an academic setting and the ones that are required by the labor market. Having this background in mind, this work aims at designing and evaluating an IoT sensor system capable of tracking patterns observed under social interactions within a group, and more specifically, in terms of the distance between group members while solving a task. Another important aspect of this study is the system's cost-effectiveness so that it can be employed in a scalable and sustainable manner. To achieve this goal, a multimethodological approach for Design Science Research was adopted, which implied the combination of several methods such as sketching, prototyping, and testing. As a result, this study contributes both to the research area of Multimodal Learning Analytics, and to educational practices.

Thesis (Ph. D.)--Aerospace Engineering, Georgia Institute of Technology, 2005.
Amy R. Pritchett, Committee Member ; Moore, Mark D., Committee Member ; Wilhite, Alan, Committee Member ; Schrage, Daniel P., Committee Chair ; Mavris, Dimitri N., Committee Co-Chair ; DeLaurentis, Daniel A., Committee Member. Vita. Includes bibliographical references.

Despite educational directives advocating the integrated use of technology, some staff remain resistant to the adoption of computer-mediated classroom practice. As resistance is sometimes akin to lack of confidence or understanding, this thesis sought to illuminate the nature of work for teachers and students in a digital environment. It reports on a descriptive and interpretive case study on the classroom experiences of teachers and senior secondary school students in two humanities subjects where technology use was integrated with literacy practices for learning (technoliteracy). Findings suggest that there is a role for technology in the learning process and that student learning might be enhanced by constructivist-based, computer-mediated activities. These findings seem to confirm the important role of teacher agency in designing classroom experiences that capitalise on the potential of new technologies of communication for effective student learning. Furthermore, this thesis has synthesised key ideas from constructivism, educational technology and learning theory with study findings to generate a supportive framework that might encourage techno-reluctant teachers to engage productively with technology in meaningful ways. It develops a grounded model for technoliteracy that gains its dynamism from the notion of design. With its related constructs of "teacher-as-designer" of classroom digital learning environment and "students-as-designers" of electronic representations of knowledge, this model suggests a purposeful integration of technology and literacy practices towards a more critical appreciation of subject content. The thesis also offers practical guidelines for applying constructivist principles to promote technoliteracy and a digital learning culture. The evaluative criteria developed from the SOLO Taxonomy and specially designed for analysing the student-created electronic artefacts also offer possibilities for reconceiving the choice of texts, activities and assessment for students of the 21st century. From the theoretical and practical perspectives shaping this study, this thesis could prompt other teachers to imagine new possibilities for digital learning and to pioneer new models for teaching and learning in increasingly borderless classrooms in the knowledge age. It could encourage techno-reluctant staff to engage in computer-mediated learning practices.

The present study explores the hypothesis that the mental representations of abstract concepts are grounded in concrete (physical) dimensions and are multimodal. Earlier studies in the framework of embodiment and conceptual metaphor suggested that the concept of power is grounded in physical size, such that big = powerful and small = powerless. In the present study, we examined if the abstract concept of power is grounded in physical size and auditory volume. Participants decided whether social groups are powerful or powerless; the group labels were presented visually, once in a big and once small font, and auditorily, once in a high and once in a low volume. Repeated measures analysis revealed that powerful words were identified more rapidly when presented in a big font or in a high volume, and powerless words when presented in a small font or in a low volume, but not all the findings were significant. Additionally, response times were predicted by item power (powerful faster) and modality (visual faster). The compatibility of the results with the hypotheses and a possible alternative explanation are discussed; item selection is considered to be the major limitation of the study.
Este estudo explora a hipótese de que as representações mentais de conceitos abstractos assentam em dimensões concretas (físicas) e são multimodais. Estudos anteriores enquadrados pelas teorias da corporalização (embodiment) e das metáforas conceptuais sugerem que o conceito de poder está assente em dimensões físicas, sendo grande = poderoso e pequeno = não poderoso. No presente estudo investigámos se o conceito abstracto de poder está assente nas dimensões de tamanho físico e volume auditivo. Os participantes classificaram grupos sociais como sendo poderosos ou não poderosos; os nomes dos grupos foram apresentados visualmente, uma vez numa fonte grande e outra vez numa fonte pequena, e auditivamente, uma vez num volume alto e outra vez num volume baixo. Análises com medidas repetidas revelaram, de acordo com o esperado, que as palavras relativas a grupos poderosos foram identificadas mais rapidamente quando apresentadas numa fonte grande ou num volume alto. Inversamente, as palavras relativas a grupos não-poderosos foram identificadas mais rapidamente quando apresentadas numa fonte pequena ou num volume baixo. No entanto, nem todos os resultados se revelaram estatisticamente significativos. Adicionalmente, os tempos de resposta dependeram do poder associado às palavras (poderosas mais rápidas) e da modalidade da apresentação (visual mais rápidas). As implicações destes resultados para as hipóteses estudadas são discutidas; a selecção das palavras é considerada como a maior limitação do estudo.

I. The aim of the dissertation The aim of this dissertation is the analysis of metaphor dynamics in multimodal mathematical discourse, as well as, establishing the extent to which conventional metaphors in mathematics become activated while solving problems in geometry. My research data consists of interviews with school teachers, academic mathematicians, students of human and exact sciences and primary school students. The interviews are devoted to the notions of area, symmetry and angle. In her work on metaphor in discourse, Cornelia Müller (2008) proposed to dispose of the binary opposition between dead and alive metaphors, proposing instead a continuum of activation from sleeping to waking metaphors. As Müller argues, the conventionality of a metaphorical expression does not imply that such a phrase does not influence the developing discourse. Her theory allows to empirically verify the influence of metaphors in discourse by grading their activation by means of signals such as, the presence of semantic contrast between the used metaphor and the discourse context, the repetition of metaphorical expressions, and the elaboration and extension of conventional metaphorical mappings. An important element of Müller’s theory is the focus on the multimodal dimensions of discourse such as gesture and visual imagery. As the researcher explains, elaborating a metaphor in more than one modality is one of the more important signals of metaphor activation. In my dissertation, I use Müller’s theory to investigate mathematical discourse, which has been an area of interest to metaphor researchers for many years. The seminal work in the field is the book by George Lakoff and Rafael Núñez Where Mathematics Comes From (2000), where the authors examined the embodied and metaphorical roots of mathematical concepts. Mathematics is a multimodal discourse realised in the form of books, lectures, graphs, and computer software. Mathematical concepts are described symbolically, verbally, graphically, and gesturally, which has been pointed out by many researchers of multimodal discourse (Edwards, 2009; Krause, 2016; Núñez, 2009; Wolfgram, 2014). I see my work as an extension and continuation of this research. With the help of Müller’s dynamic view, and in line with the work of linguists interested in multimodal studies of mathematics, I am trying to show how geometrical concepts emerge through the activation of metaphor on the level of speech, gesture, and visual imagery. By means of case studies based on interviews on geometrical concepts, I verify the possibility of activating conventional metaphors in mathematical discourse and discuss whether the conventional metaphors of mathematics become active for the purpose of explaining and solving problems. II. The significance of this research project The topic of metaphors in the language of science and mathematics has been an area of linguistic research for some time, yet apart from Lakoff and Núñez’s work, not many studies have considered the dynamics and emergence of mathematical concepts in discourse. In recent years, this area of enquiry was explored by didacticians interested in the role of gesture, instruments and drawings in the creation of mathematical concepts (de Freitas & Sinclair, 2014; Roth, 2011). The multimodal and dynamic approach towards metaphor elaborates on this research area from a linguistic perspective and allows for the analysis of the role of various communication dimensions in the emergence of concepts. My project also fits into the relatively new research area of discourse dynamics (Cameron et al., 2009), providing new insights on the basis of independently collected data. III. Research methodology The collected data consists of 6 lengthy interviews with teachers, academic mathematicians and students of different disciplines. The interviews explore the notions of area, symmetry and angle, along with some related geometrical problems. Additionally, the research material is supplemented by a series of short interviews with students collected during the meetings of a primary school mathematics club. These interviews discuss the concept of area. In my analyses I use the methodology proposed by Müller (2008), and other researchers of multimodality (Antas, 2013; Cienki, 2010; Forceville, 2008; McNeill, 1992, 2005; Mittelberg, 2007, 2008, 2013; Mittelberg & Waugh, 2009; Müller, 1998). The interviews were video-recorded, transcribed and annotated using EXMARaLDA (Schmidt and Wörner, 2009), a software package which helps in the synchronic analysis of speech and gesture. My analyses also consider the drawings created by the participants during the interviews. IV. Structure of the dissertation and conclusions The dissertation consists of four main chapters. The first discusses the notion of metaphor and its aspects relevant to my work. Here, I explain the topics of multimodal metaphor and metaphors in mathematics. I also describe Müller’s dynamic view and explore the idea of metaphor emergence in discourse. The second chapter is devoted to the role of metaphors in mathematics. It is based on academic work on scientific models (Black, 1962; Zawisławska, 2014), as well as historical and philosophical works on the development of mathematics (Châtelet, 2000; Rotman, 1993, 2000). The third chapter is a detailed discussion of my research methodology. The fourth chapter presents an analysis of the conducted interviews. Here, I discuss metaphors related to area, units of measure, different symmetry types, geometrical transformations, transformation invariants, and the static and dynamic understanding of angles. The following ideas are discussed in the conclusion to my dissertation: • Concepts in mathematics discourse are built around the interaction of multiple modalities, especially in the form of multimodal metaphors. • Metaphors in mathematics discourse are dynamic and they emerge over time. • Active domains have a persistent influence over a discourse event. • To explore deep aspects of target concepts, speakers either elaborate on their metaphors, or employ new ones which complement one another. • Metaphors organise mathematics discourse by becoming the point of reference for participants. • Metaphors in geometry are grounded in physical experience.
I. Cel naukowy rozprawy Celem rozprawy jest analiza dynamiki metafory w dyskursie multimodalnym na przykładzie matematyki oraz sprawdzenie w jakim stopniu konwencjonalne metafory na temat pojęć matematycznych ulegają aktywacji podczas rozwiązywania problemów geometrycznych. Za materiał badawczy posłużyły mi wywiady przeprowadzone z nauczycielami oraz wykładowcami matematyki, studentami kierunków humanistycznych i ścisłych a także z uczniami szkoły podstawowej. Wywiady te omawiają trzy wybrane pojęcia geometryczne: pole powierzchni, symetria, kąt. W swoich badaniach nad metaforą w dyskursie Cornelia Müller (2008) zaproponowała odrzucenie binarnego podziału na metafory żywe i martwe proponując w zamian kontinuum aktywacji od metafor uśpionych do obudzonych. Jak uzasadnia Müller, sama konwencjonalność wyrażenia metaforycznego nie jest jednoznaczną oznaką tego że dana metafora pojawia się jedynie jako utarty zwrot bez znaczenia dla kształtu rozwijającego się dyskursu. Teoria Müller pozwala empirycznie zweryfikować wpływ metafor na przebieg dyskursu stopniując ich aktywację poprzez takie oznaki jak: semantyczny kontrast użytej metafory w stosunku do reszty dyskursu; powtarzanie danej metafory; jak również rozbudowanie użytych metafor poprzez rozszerzenie i uszczegółowienie konwencjonalnych rzutowań metaforycznych. Istotnym elementem teorii Müller jest zwrócenie uwagi na wymiary dyskursu wychodzące poza modalność werbalną, takie jak gest czy obraz. Jak uzasadnia badaczka, rozciągnięcie się danej metafory na wiele modalności to jeden z istotnych wskaźników aktywowanych metafor. W mojej rozprawie wykorzystuję teorię Müller do badania dyskursu matematycznego, który od wielu lat leży w kręgu zainteresowań lingwistów zajmujących się metaforą pojęciową. Kluczową pracą w tej dziedzinie była książka Georga Lakoffa i Rafaela Núñeza Where Mathematics Comes From (2000), gdzie badacze wskazywali na cielesne (ang. embodied) i metaforyczne podwaliny pojęć matematycznych. Matematyka jest dyskursem multimodalnym realizującym się w formie książek, wykładów, wykresów i programów komputerowych. Pojęcia matematyczne opisywane są symbolicznie, słownie, graficznie oraz gestycznie, co nie umknęło badaczom dyskursu multimodalnego (Edwards, 2009; Krause, 2016; Núñez, 2009; Wolfgram, 2014). Własną pracę postrzegam jako rozwinięcie i kontynuację tych badań. Wykorzystując dynamiczną teorię Müller oraz opierając się o doświadczenia językoznawców w multimodalnych studiach nad matematyką, staram się pokazać jak poprzez aktywację metafor na płaszczyźnie werbalnej, gestycznej i obrazowej dokonuje się wyłanianie pojęć geometrycznych. Studia przypadków oparte na wywiadach dotyczących pojęć matematycznych pozwalają zweryfikować możliwość aktywacji konwencjonalnych metafor i odpowiedzą na pytanie czy metafory w matematyce budzą się na potrzeby wyjaśniania i rozwiązywania problemów. II. Znaczenie prowadzonych badań Tematyka metafor w języku nauki i matematyki przewija się w literaturze językoznawczej już od pewnego czasu. Choć metafory matematyczne doczekały się tak systematycznego opracowania, jak w pracy Lakoffa i Núñeza, to wciąż mało miejsca poświęcono dynamice wyłaniania się pojęć matematycznych w dyskursie. Temat ten w ostatnich latach pojawiał się literaturze z zakresu dydaktyki (de Freitas i Sinclair, 2014; Roth, 2011), w której badacze zwracają szczególną uwagę na role narzędzi, rysunków oraz gestów podczas wytwarzania pojęć. Multimodalne i dynamiczne podejście do metafory wydaje się naturalnym uzupełnieniem tego obszaru badań z perspektywy językoznawczej i pozwala na przeanalizowanie interakcji różnych poziomów komunikacji w budowaniu pojęć matematycznych. Prowadzone przeze mnie badania wpisują się również w stosunkowo nowy nurt dynamiki dyskursu (Cameron i in., 2009) i uzupełniają dziedzinę o samodzielnie zebrany materiał badawczy. III. Metodyka badań Materiał badawczy składa się z 6 długich wywiadów, z nauczycielami matematyki, studentami różnych dyscyplin i wykładowcami matematyki. Wywiady obejmują omawianie zagadnień geometrycznych i rozwiązywanie prostych problemów dotyczących pojęć pola powierzchni, symetrii i kąta. Dodatkowo, materiał badawczy został rozbudowany o krótkie wywiady z uczniami szkoły podstawowej przeprowadzone w trakcie zajęć koła matematycznego na temat pojęcia pola powierzchni. W analizie danych kieruję się metodologią badawczą Müller (2008), a także pracami badaczy multimodalności (Antas, 2013; Cienki, 2010; Forceville, 2008; McNeill, 1992, 2005; Mittelberg, 2007, 2008, 2013; Mittelberg i Waugh, 2009; Müller, 1998). Wywiady zostały zarejestrowane przy pomocy kamery a następnie przetranskrybowane i anotowane w programie EXMARaLDA (Schmidt i Wörner, 2009) ułatwiającym synchroniczne badanie mowy i gestów. W badaniu wzięte pod uwagę są również rysunki tworzone przez uczestników w trakcie rozmów. IV. Struktura rozprawy i wnioski Rozprawa składa się z czterech głównych rozdziałów. Pierwszy omawia pojęcie metafory i jego aspekty istotne dla dalszej części pracy. Poświęcam tu miejsce przede wszystkim metaforze w dyskursie multimodalnym a także matematycznym. Charakteryzuję też dynamiczne podejście Müller i omawiam zagadnienia dotyczące wyłaniania się metafor w dyskursie. W drugim rozdziale omawiam rolę metafor w matematyce w oparciu o literaturę poświęconą modelom naukowym (Black, 1962; Zawisławska, 2014) oraz historyczne i filozoficzne prace na temat rozwoju matematyki (Châtelet, 2000; Rotman, 1993, 2000). Trzeci rozdział szczegółowo omawia metodologię badawczą. Na czwarty rozdział składa się analiza przeprowadzonych wywiadów. Omawiam tutaj metafory związane z polem powierzchni, jednostkami pola, rodzajami symetrii, przekształceniami geometrycznymi i ich niezmiennikami, jak również statyczne i dynamiczne rozumienie pojęcia kąta. W podsumowaniu wyciągam wnioski z części badawczej pracy: • Pojęcia w matematyce budowane są w poprzez interakcje różnych modalności ze szczególnym uwzględnieniem metafor multimodalnych. • Metafory w dyskursie matematycznym są dynamiczne i wyłaniają się w czasie. • Aktywne domeny oddziałują na dyskurs w sposób długotrwały. • Analiza pojęć matematycznych w dyskursie odbywa się poprzez rozbudowanie pojedynczych metafor lub proponowanie kilku metafor uzupełniających się wzajemnie. • Metafory organizują dyskurs matematyczny poprzez bycie punktami odniesienia dla uczestników dyskursu. • Metafory w geometrii ugruntowane są w fizycznym doświadczaniu świata.

This thesis deals with the development of exploratory concepts of multimodal interactions able to support the Human Performance (HP) of Tower (TWR) Air Traffic Controllers (ATCOs) from a remote tower, during the traffic management on the area under their responsibility. Nowadays the TWR Controllers’ job consists in the management of the aircraft and vehicles movements on the runways and taxiways of an airport and they perform their tasks with the Out-The-Window (OTW) view, which means that they look at the vehicles without using any screens, but directly looking outside of the tower building windows, especially in small and medium airports. In recent years, a new operational concept emerged in aviation: the Remote Tower Operations (RTOs). The motivation of the growth of this new concept is the current European situation of small and medium density airports that often have to face the difficulties and the high costs for the provision of a safe and efficient Air Traffic Service (ATS). The idea behind is that in the new remote working environment, the Tower ATCO will no longer perform his/her tasks from a local aerodrome but from a distant position. According to the European guidelines that are being developed to define the technical and operational requirements of the Remote Tower system, it is supposed that the TWR ATCOs will monitor the traffic mainly through the implementation of high definition cameras, including also radar screens and radio transmission, instead of the OTW. In addition to the visual information, the aerodrome outdoor sound is also taken into consideration for the development of the Control Working Position (CWP) support tools, without any mention to data collected through other sensory modalities. The PhD work has been carried out in the framework of a European project called MOTO (The embodied reMOte TOwer) which, considering the above-mentioned context and the theories of multisensory interaction and embodied cognition, explores the possibility to improve human performance in RTOs. The multisensory concept and the embodied cognition, which study the feeling and the modalities humans use to interact with the environment, have been considered as the theoretical context to achieve a full understanding on how controllers use all of their senses during every day operations, in order to improve the situation awareness and to decrease the workload. This approach offers the advantage of analysing complex human behaviours during natural interactions in real-world environments and in enriched simulations of the real world.

In the present work, an in-depth exploration of IMAGACT Ontology of Action Verbs has been traced, with the focus of exploiting the resource in NLP tasks. Starting from the Introduction, the idea of making use of IMAGACT multimodal action conceptualisation has been drawn, with some reflections on evidences of the deep linking between Language and Vision, and on the fact that action plays a key role in this linkage. Thus, the multimodal and multilingual features of IMAGACT have been described, with also some details on the framework of the resource building. It followed a concrete case-study on IMAGACT internal data, that led to the proposal of an inter-linguistic manual mapping between the Action Types of verbs which refer to cutting eventualities in English and Italian. Then, a series of ex-periments have been presented, involving the exploitation of IMAGACT in linking with other resources and building deliverable NLP products (such as the Ref-vectors of action verbs). One of the experiments has been described extensively: the visual enrichment of IMAGACT through instance population of its action concepts, making use of Audio Description of movies for visually impaired people. From this last experiment it emerged that dealing with non-conventional scenarios, such as the one of assessing action reference similarity between texts from different domains, is particularly challenging, given that fine-grained differences among action concepts are difficult to derive purely from the textual representation.

碩士
淡江大學
日本語文學系碩士班
101
The purpose of this paper is to explore the characteristics of expressional structures and appeal concepts in Japanese TV advertising, and examine them from the multi-modal perspective in verbal and non-verbal communications. The subjects were 71 award-winning TV advertisements in Japan from 2008 through to 2012 in three different fields. The results of the examination were as follow. 　　Based on the various elements that made up the TV advertising, advertising slogans that did not specifically mention the product, define the product niche in an indirect manner and focus on constructing the "association-ambience "type of the brand image appeared with greatest frequency. Such an approach can be considered representative of the advertising technique used during Japan''s economic stagnation. For the characteristics of theme presentation and placement, these could be divided into the "middle" type, where only the product appeared or was mentioned in the verbal communication, as well as the "conclusion" type where the main message appeared only at the end of the advertisement. In other words, most products featured some form of introductory expression before getting to the main subject. Most also chose not to explicitly reveal or refer to the product at the same time. As for the image characteristics, over 70％ of the advertising was filmed at a steady pace to convey the message and construct brand image. 　　Examination of appeal concepts found that approximately 37％ of the advertising concepts where of the stand-alone type, while nearly 63％ were of the composite type. Stand-alone advertisements appeared most frequently as the "lifestyle type" that used events from the consumer''s everyday life to be more engaging. For composite advertising, the main character''s personal charisma and nature were used to enhance the consumer''s affinity for the product or brand, and combined with the "humor ＆ personality symbol type" where humorous elements were added for consumer appeal, and appeared with the greatest frequency. 　　The results of the examination presented above showed that the subjects of this paper tended towards the following expressions. First, everyday events were used as the background, and the main subject was usually mentioned only after the introductory expression. The verbal communication was generally rich in implications and did not expressly mention the product. The main character''s personal charisma and nature were also drawn upon to enhance consumer affinity for a product or brand. Humorous elements were then added to appeal to the consumer. Together, these formed the most common type of appeal expression.

Cette thèse considère l'apprentissage de motifs récurrents dans la perception multimodale. Elle s'attache à développer des modèles robotiques de ces facultés telles qu'observées chez l'enfant, et elle s'inscrit en cela dans le domaine de la robotique développementale. Elle s'articule plus précisément autour de deux thèmes principaux qui sont d'une part la capacité d'enfants ou de robots à imiter et à comprendre le comportement d'humains, et d'autre part l'acquisition du langage. À leur intersection, nous examinons la question de la découverte par un agent en développement d'un répertoire de motifs primitifs dans son flux perceptuel. Nous spécifions ce problème et établissons son lien avec ceux de l'indétermination de la traduction décrit par Quine et de la séparation aveugle de source tels qu'étudiés en acoustique. Nous en étudions successivement quatre sous-problèmes et formulons une définition expérimentale de chacun. Des modèles d'agents résolvant ces problèmes sont également décrits et testés. Ils s'appuient particulièrement sur des techniques dites de sacs de mots, de factorisation de matrices et d'apprentissage par renforcement inverse. Nous approfondissons séparément les trois problèmes de l'apprentissage de sons élémentaires tels les phonèmes ou les mots, de mouvements basiques de danse et d'objectifs primaires composant des tâches motrices complexes. Pour finir nous étudions le problème de l'apprentissage d'éléments primitifs multimodaux, ce qui revient à résoudre simultanément plusieurs des problèmes précédents. Nous expliquons notamment en quoi cela fournit un modèle de l'ancrage de mots acoustiques.

With recent advances in computer vision, researchers have been able to demonstrate impressive performance at near-human-level capabilities in difficult tasks such as image recognition. For example, for images taken under typical conditions, computer vision systems now have the ability to recognize if a dog, cat, or car appears in an image. These advances are made possible by utilizing the massive volume of image datasets and label annotations, which include category labels and sometimes bounding boxes around the objects of interest within the image. However, one major limitation of the current solutions is that when users apply recognition models to new domains, users need to manually define the target classes and label the training data in order to prepare labeled annotations required for the process of training the recognition models. Manually identifying the target classes and constructing the concept ontology for a new domain are time-consuming tasks, as they require the users to be familiar with the content of the image collection, and the manual process of defining target classes is difficult to scale up to generate a large number of classes. In addition, there has been significant interest in developing knowledge bases to improve content analysis and information retrieval. Knowledge base is an object model (ontology) with classes, subclasses, attributes, instances, and relations among them. The knowledge base generation problem is to identify the (sub)classes and their structured relations for a given domain of interest. Similar to ontology construction, Knowledge base is usually generated by human experts manually, and it is usually a time-consuming and difficult task. Thus, it is important and necessary to find a way to explore the semantic concepts and their structural relations that are important for a target data collection or domain of interest, so that we can construct an ontology or knowledge base for visual data or multimodal content automatically or semi-automatically. Visual patterns are the discriminative and representative image content found in objects or local image regions seen in an image collection. Visual patterns can also be used to summarize the major visual concepts in an image collection. Therefore, automatic discovery of visual patterns can help users understand the content and structure of a data collection and in turn help users construct the ontology and knowledge base mentioned earlier. In this dissertation, we aim to answer the following question: given a new target domain and associated data corpora, how do we rapidly discover nameable content patterns that are semantically coherent, visually consistent, and can be automatically named with semantic concepts related to the events of interest in the target domains? We will develop pattern discovery methods that focus on visual content as well as multimodal data including text and visual. Traditional visual pattern mining methods only focus on analysis of the visual content, and do not have the ability to automatically name the patterns. To address this, we propose a new multimodal visual pattern mining and naming method that specifically addresses this shortcoming. The named visual patterns can be used as discovered semantic concepts relevant to the target data corpora. By combining information from multiple modalities, we can ensure that the discovered patterns are not only visually similar, but also have consistent meaning, as well. The capability of accurately naming the visual patterns is also important for finding relevant classes or attributes in the knowledge base construction process mentioned earlier. Our framework contains a visual model and a text model to jointly represent the text and visual content. We use the joint multimodal representation and the association rule mining technique to discover semantically coherent and visually consistent visual patterns. To discover better visual patterns, we further improve the visual model in the multimodal visual pattern mining pipeline, by developing a convolutional neural network (CNN) architecture that allows for the discovery of scale-invariant patterns. In this dissertation, we use news as an example domain and image caption pairs as example multimodal corpora to demonstrate the effectiveness of the proposed methods. However, the overall proposed framework is general and can be easily extended to other domains. The problem of concept discovery is made more challenging if the target application domain involves fine-grained object categories (e.g., highly related dog categories or consumer product categories). In such cases, the content of different classes could be quite similar, making automatic separation of classes difficult. In the proposed multimodal pattern mining framework, representation models for visual and text data play an important role, as they shape the pool of candidates that are fed to the pattern mining process. General models like the CNN models trained on ImageNet, though shown to be generalizable to various domains, are unable to capture the small differences in the fine-grained dataset. To address this problem, we propose a new representation model that uses an end-to-end artificial neural network architecture to discover visual patterns. This model can be fine-tuned on a fine-grained dataset so that the convolutional layers can be optimized to capture the features and patterns from the fine-trained image set. It has the ability to discover visual patterns from fine-grained image datasets because its convolutional layers of the CNN can be optimized to capture the features and patterns from the fine-grained images. Finally, to demonstrate the advantage of the proposed multimodal visual pattern mining and naming framework, we apply the proposed technique to two applications. In the first application, we use the visual pattern mining technique to find visual anchors to summarize video news events. In the second application, we use the visual patterns as important cues to link video news events to social media events. The contributions of this dissertation can be summarized as follows: (1) We develop a novel multimodal mining framework for discovering visual patterns and nameable concepts from a collection of multimodal data and automatically naming the discovered patterns, producing a large pool of semantic concepts specifically relevant to a high-level event. The framework combines visual representation based on CNN and text representation based on embedding. The named visual patterns can be required for construct event schema needed in the knowledge base construction process. (2) We propose a scale-invariant visual pattern mining model to improve the multimodal visual pattern mining framework. The improved visual model leads to better overall performance in discovering and naming concepts. To localize the visual patterns discovered in this framework, we propose a deconvolutional neural network model to localize the visual pattern patterns within the image. (3) To directly learn from data in the target domain, we propose a novel end-to-end neural network architecture called PatternNet for finding high-quality visual patterns even for datsets that consistent of fine-grained classes. (4) We demonstrate novel applications of visual pattern mining in two applications: video news event summarization and video news event linking.

Since the Second World War fiber reinforced polymer (FRP) composites have become more attractive as a structural material in a variety of engineering practices, such as infrastructure construction, automobile industry and aerospace engineering, due to high specific strength and stiffness as well as flexibility in tailoring the structural performance. On the other hand, a stiffened panel always performs better in resisting loads compared to an unstiffened panel of same weight. Thus a combination of lightweight composite materials and stiffened structural forms can efficiently enhance the load resisting capability that can be buckling strength of a structure. Stiffened composite panels are subjected to any combination of in-plane, out-of- plane and shear load conditions during service life. These types of thin-walled structures are vulnerable to lose global and local stability under compression loadings. Consequently, buckling-resistant design is one of the most critical issues of stiffened composite panels applied in real practices. Moreover, the buckling optimization design of composite panels is usually a typical multimodal optimization problem, in which there exist multiple global optimal solutions with identical or closely comparable optima of structural performance. Recently, with the development of manufacturing techniques, advanced grid stiffened (AGS) composite panels have increasingly emerged and gained more attention as these grid-stiffening configurations help to enhance the structural efficiency in a more effective way in complex loading conditions compared with conventional unidirectionally-stiffened composite panels. These grid-stiffening configurations provide more available options to select outstanding concept designs of AGS composite panels against buckling for the final appropriate design development at the final construction stage. In this PhD thesis, a novel multi-level optimization methodology for concept design of advanced grid stiffened composite panels against buckling has been developed. Furthermore, an efficient finite element (FE) modelling component for buckling analysis and a robust particle swarm optimization (PSO) algorithms for multimodal optimization have been presented, in order to further consolidate the performance of the proposed methodology. The thesis is divided into six chapters, which are briefly described below: In Chapter 1, a general background along with the objective and originality of the present research is presented. An efficient FE modelling technique is presented in Chapter 2, for the prediction of buckling response of grid stiffened composite panels having different stiffening arrangements. The laminated skin of the stiffened structure is modelled with a triangular degenerated curved shell element. An efficient curved beam element compatible with the shell element is developed for the modelling of stiffeners which may have different lamination schemes. The deformation of the beam element is completely defined in terms of the degrees of freedom of shell elements and it does not require any additional degrees of freedom. Chapter 3 aims to extend conventional unimodal optimization to challenging multimodal optimization of composite structures, by means of newly emerged multimodal PSO using niching techniques. It has shown that the ring topology based PSO without any niching parameter is more robust and efficient for multimodal optimization of composite structures, compared with the species-based PSO (SPSO) and the fitness Euclidean-distance ratio based PSO (FERPSO). In Chapter 4, a random reflection boundary is proposed to replace the conventional fixed absorption boundary for the range-exceeding particles, in order to eliminate/reduce the significance and sensitivity of an empirical parameter of particles’ maximum velocity in PSO. Based on the results obtained from the experimentation on the abovementioned test functions, empirical guidelines for appropriately using the half-range/full-range random reflection boundary are further proposed. Chapter 5 presents an efficient methodology to conduct concept design of AGS composite panels, based on a multi-level approach where an inner 3- stage optimization process is nested within an outer 3-step optimization process. The proposed methodology is applied to a design optimization problem of an AGS composite plate against its buckling resistance, by incorporating a ring topology based multimodal PSO algorithm with an improved FE buckling analysis model. Finally, the conclusions of the present research are summarized in Chapter 6. The limitations and the future development directions of the present study are also described in this chapter.
Thesis (Ph.D.) -- University of Adelaide, School of Civil, Environmental and Mining Engineering, 2015

Tese de mestrado, Engenharia Informática (Engenharia de Software) Universidade de Lisboa, Faculdade de Ciências, 2020
Atualmente, a tecnologia encontra-se cada vez mais presente no nosso quotidiano, sendo que alguns dispositivos são indispensáveis, uma vez que permitem a utilização de ferramentas que nos ajudam na nossa organização pessoal, social e profissional. Para o desenvolvimento de dispositivos e ferramentas, são necessários conhecimentos de programação que são utilizados na produção de algoritmos e na criação de sistemas. A aprendizagem de conceitos de programação não é fácil, tornando-se ainda mais difícil para pessoas, em que o acesso a estes conteúdos é mais limitado, nomeadamente a pessoas cegas ou com baixa visão. Existe, por este motivo, uma grande necessidade de tornar a sua aprendizagem mais simples e acessível. Estudos, efetuados no passado revelam que a aquisição destes conhecimentos em idades muito precoces possibilita a sua compreensão de uma forma mais eficaz. Uma das contribuições deste trabalho foi desenvolver um sistema que permite a aprendizagem de conceitos de programação a crianças com deficiências visuais, através da utilização de tangíveis ou do reconhecimento de voz, para que um robô se movimente num mapa construído com Legos. O sistema possibilita que sejam construídos caminhos com peças de diferentes cores, permitindo o seu mapeamento por uma aplicação que pode ser executada num tablet ou smartphone. Esta tem a capacidade de obter sequências através de blocos tangíveis em que são colocadas peças, que contêm relevos permitindo a identificação táctil por crianças cegas, ou através de reconhecimento por voz, no qual a aplicação faz várias questões sobre o que o utilizador pretende fazer, sendo que este ao responder vai construindo o algoritmo. O robô de pequenas dimensões executa a sequência efetuada atravessando o caminho construído.
Currently, technology is increasingly present in our daily lives, and some devices are indispensable, since they allow the use of tools that help us in our personal, social and professional organization. For the creation of devices and tools, knowledge of programming is required, for the production of algorithms that lead to the creation of these. Learning programming concepts is not easy, making it even more difficult for people, where access to these contents is more complicated, such as blind and low vision people. For this reason, there is a great need to make learning more simple and accessible for people with visual impairments. Some studies carried out show that the acquisition of this knowledge at very early ages makes it possible to understand it more effectively. One of the contributions of this work was to develop a system, which allows the learning of programming concepts to children with visual impairments through the use of tangibles or voice recognition so that a robot moves on a map built with Legos. The system allows paths to be built with pieces of different colors, allowing their recognition by an application that can be run on a tablet or smartphone. It also allows obtaining sequences through tangible blocks in which pieces containing reliefs are placed allowing tactile identification by blind children, or through voice recognition, in which the application asks several questions about what the user can do and that the user responding builds the algorithm. The small robot executes the sequence carried out across the constructed path.

The main question of this study was: How and in what way can a multimodal semiotic visual analysis model be developed and used for contributing to the analysis and understanding of the manner in which the Men’s Health (South Africa) magazine – as a case study – represents and constructs masculinities in South Africa? The following three subsidiary research questions were formulated to address this topic: • What is the literature revealing with reference to the media as producers of meaning in relation to masculinity and visual texts? • How and in which way can a semiotic visual analysis multimodal model be developed with the purpose of contributing to the analysis of visual texts? • What is the outcome of the visual analysis multimodal model with reference to the case study about the representation and construction of masculinities in visual texts in MH? The first aim of this research was to establish an overview of masculinities and to explore the visual representation of masculinity with reference to mediation, reality, and ideology in the media. With reference to the media as producers of meaning in relation to masculinity and visual texts, a semiotic visual analysis and social semiotics were used to unpack culture as a site of the production of meanings. The media is one of the main sources from which men receive their entertainment and information about the world. In this sense, the media makes sense of the world. Mass media plays a key role in discourse and constructing the relationships between reality and ideology. During this construction, the media reflects on existing opinions and attitudes in society. A quantitative content analysis and a qualitative semiotic multimodal visual analysis were conducted on 27 visual texts purposively selected from MH to include editions from July 2010 to June 2011. This population covered 12 front covers, 12 editorials and three flip covers. The developed visual multimodal model was tested qualitatively on nine visual texts since these texts included the front covers, flip covers and editorials of the three editions with flip covers. v A second major aim of the study was to establish the way in which a semiotic visual analysis multimodal model needed to be developed and used for analysing visual texts, as well as for analysing the visual texts according to the multimodal model in order to understand how the multimodality and social semiotic resources were applied in MH to represent and construct masculinities. The rationale for the development and design of this model was based on the premise that a basic understanding of semiotics and visual language was needed. Without such an understanding, the vast amounts of visual messages that confront the reader would remain incomprehensible. Consequently, a productive dialogue in relation to visual communication cannot take place. The multimodal model developed in this thesis highlights visual text layout, in conjunction with language-in-use, that does not occur in isolation and that is deeply reliant on other forms of making meaning. The heptagon multimodal model consists of concept maps of the six functions of the designed hexagon model. This multimodality approach includes analysing simultaneously occurring semiotics and their various roles in conjunction with detailed, all-inclusive discourses. In the quantitative content analysis and the qualitative multimodal semiotic analysis, the six components of the developed heptagon model (visual grammar, positioning, typography, colour, modality, and iconography) are illustrated. The quantitative research supported the main research design, i.e. the qualitative multimodal semiotic analysis. It is envisaged that the development and construction of a multimodal semiotic model will make a contribution to the scholarly field of semiotic analysis. By discussing the fluidity of the variations of masculinities and male identities, by giving a brief overview of the role of the media in constructing masculinities, and by focusing on the discourses that took place in MH, the researcher creates an awareness of the inherited patriarchal masculinities by recommending envisioned masculinities to be inclusive as a component of the solution. This approach is illustrated by the use and findings of the multimodal semiotic visual analysis.
Communication Science
D. Litt. et. Phil