Добірка наукової літератури з теми "Multimodal object detection"

Detecting objects on unmanned aerial vehicles is a hard task, due to the long visual distance and the subsequent small size and lack of view. Besides, the traditional ground observation manners based on visible light camera are sensitive to brightness. This article aims to improve the target detection accuracy in various weather conditions, by using both visible light camera and infrared camera simultaneously. In this article, an association network of multimodal feature maps on the same scene is used to design an object detection algorithm, which is the so-called feature association learning method. In addition, this article collects a new cross-modal detection data set and proposes a cross-modal object detection algorithm based on visible light and infrared observations. The experimental results show that the algorithm improves the detection accuracy of small objects in the air-to-ground view. The multimodal joint detection network can overcome the influence of illumination in different weather conditions, which provides a new detection means and ideas for the space-based unmanned platform to the small object detection task.

To construct a safe and sound autonomous driving system, object detection is essential, and research on fusion of sensors is being actively conducted to increase the detection rate of objects in a dynamic environment in which safety must be secured. Recently, considerable performance improvements in object detection have been achieved with the advent of the convolutional neural network (CNN) structure. In particular, the YOLO (You Only Look Once) architecture, which is suitable for real-time object detection by simultaneously predicting and classifying bounding boxes of objects, is receiving great attention. However, securing the robustness of object detection systems in various environments still remains a challenge. In this paper, we propose a weighted mean-based adaptive object detection strategy that enhances detection performance through convergence of individual object detection results based on an RGB camera and a LiDAR (Light Detection and Ranging) for autonomous driving. The proposed system utilizes the YOLO framework to perform object detection independently based on image data and point cloud data (PCD). Each detection result is united to reduce the number of objects not detected at the decision level by the weighted mean scheme. To evaluate the performance of the proposed object detection system, tests on vehicles and pedestrians were carried out using the KITTI Benchmark Suite. Test results demonstrated that the proposed strategy can achieve detection performance with a higher mean average precision (mAP) for targeted objects than an RGB camera and is also robust against external environmental changes.

Visual relationship detection aims to completely understand visual scenes and has recently received increasing attention. However, current methods only use the visual features of images to train the semantic network, which does not match human habits in which we know obvious features of scenes and infer covert states using common sense. Therefore, these methods cannot predict some hidden relationships of object-pairs from complex scenes. To address this problem, we propose unifying vision–language fusion and knowledge graph reasoning to combine visual feature embedding with external common sense knowledge to determine the visual relationships of objects. In addition, before training the relationship detection network, we devise an object–pair proposal module to solve the combination explosion problem. Extensive experiments show that our proposed method outperforms the state-of-the-art methods on the Visual Genome and Visual Relationship Detection datasets.

The object detection algorithm is a key component for the autonomous operation of unmanned surface vehicles (USVs). However, owing to complex marine conditions, it is difficult to obtain large-scale, fully labeled surface object datasets. Shipborne sensors are often susceptible to external interference and have unsatisfying performance, compromising the results of traditional object detection tasks. In this paper, a few-shot surface object detection method is proposed based on multimodal sensor systems for USVs. The multi-modal sensors were used for three-dimensional object detection, and the ability of USVs to detect moving objects was enhanced, realizing metric learning-based few-shot object detection for USVs. Compared with conventional methods, the proposed method enhanced the classification results of few-shot tasks. The proposed approach achieves relatively better performance in three sampled sets of well-known datasets, i.e., 2%, 10%, 5% on average precision (AP) and 28%, 24%, 24% on average orientation similarity (AOS). Therefore, this study can be potentially used for various applications where the number of labeled data is not enough to acquire a compromising result.

Unpaired image-to-image translation is proven quite effective in boosting a CNN-based object detector for a different domain by means of data augmentation that can well preserve the image-objects in the translated images. Recently, multimodal GAN (Generative Adversarial Network) models have been proposed and were expected to further boost the detector accuracy by generating a diverse collection of images in the target domain, given only a single/labelled image in the source domain. However, images generated by multimodal GANs would achieve even worse detection accuracy than the ones by a unimodal GAN with better object preservation. In this work, we introduce cycle-structure consistency for generating diverse and structure-preserved translated images across complex domains, such as between day and night, for object detector training. Qualitative results show that our model, Multimodal AugGAN, can generate diverse and realistic images for the target domain. For quantitative comparisons, we evaluate other competing methods and ours by using the generated images to train YOLO, Faster R-CNN and FCN models and prove that our model achieves significant improvement and outperforms other methods on the detection accuracies and the FCN scores. Also, we demonstrate that our model could provide more diverse object appearances in the target domain through comparison on the perceptual distance metric.

Multiobject Tracking (MOT) is one of the most important abilities of autonomous driving systems. However, most of the existing MOT methods only use a single sensor, such as a camera, which has the problem of insufficient reliability. In this paper, we propose a novel Multiobject Tracking method by fusing deep appearance features and motion information of objects. In this method, the locations of objects are first determined based on a 2D object detector and a 3D object detector. We use the Nonmaximum Suppression (NMS) algorithm to combine the detection results of the two detectors to ensure the detection accuracy in complex scenes. After that, we use Convolutional Neural Network (CNN) to learn the deep appearance features of objects and employ Kalman Filter to obtain the motion information of objects. Finally, the MOT task is achieved by associating the motion information and deep appearance features. A successful match indicates that the object was tracked successfully. A set of experiments on the KITTI Tracking Benchmark shows that the proposed MOT method can effectively perform the MOT task. The Multiobject Tracking Accuracy (MOTA) is up to 76.40% and the Multiobject Tracking Precision (MOTP) is up to 83.50%.

As an important field of computer vision, object detection has been studied extensively in recent years. However, existing object detection methods merely utilize the visual information of the image and fail to mine the high-level semantic information of the object, which leads to great limitations. To take full advantage of multi-source information, a knowledge update-based multimodal object recognition model is proposed in this paper. Specifically, our method initially uses Faster R-CNN to regionalize the image, then applies a transformer-based multimodal encoder to encode visual region features (region-based image features) and textual features (semantic relationships between words) corresponding to pictures. After that, a graph convolutional network (GCN) inference module is introduced to establish a relational network in which the points denote visual and textual region features, and the edges represent their relationships. In addition, based on an external knowledge base, our method further enhances the region-based relationship expression capability through a knowledge update module. In summary, the proposed algorithm not only learns the accurate relationship between objects in different regions of the image, but also benefits from the knowledge update through an external relational database. Experimental results verify the effectiveness of the proposed knowledge update module and the independent reasoning ability of our model.

Abstract. Object Re-Identification (ReID) is the task of matching a given object in the new environment with its image captured in a different environment. The input for a ReID method includes two sets of images. The probe set includes one or more images of the object that must be identified in the new environment. The gallery set includes images that may contain the object from the probe image. The ReID task’s complexity arises from the differences in the object appearance in the probe and gallery sets. Such difference may originate from changes in illumination or viewpoint locations for multiple cameras that capture images in the probe and gallery sets. This paper focuses on developing a deep learning ThermalReID framework for cross-modality object ReID in thermal images. Our framework aims to provide continuous object detection and re-identification while monitoring a region from a UAV. Given an input probe image captured in the visible range, our ThermalReID framework detects objects in a thermal image and performs the ReID. We evaluate our ThermalReID framework and modern baselines using various metrics. We use the IoU and mAP metrics for the object detection task. We use the cumulative matching characteristic (CMC) curves and normalized area-under-curve (nAUC) for the ReID task. The evaluation demonstrated encouraging results and proved that our ThermalReID framework outperforms existing baselines in the ReID accuracy. Furthermore, we demonstrated that the fusion of the semantic data with the input thermal gallery image increases the object detection and localization scores. We developed the ThermalReID framework for cross-modality object re-identification. We evaluated our framework and two modern baselines on the task of object ReID for four object classes. Our framework successfully performs object ReID in the thermal gallery image from the color probe image. The evaluation using real and synthetic data demonstrated that our ThermalReID framework increases the ReID accuracy compared to modern ReID baselines.

The stabilization and validation process of the measured position of objects is an important step for high-level perception functions and for the correct processing of sensory data. The goal of this process is to detect and handle inconsistencies between different sensor measurements, which result from the perception system. The aggregation of the detections from different sensors consists in the combination of the sensorial data in one common reference frame for each identified object, leading to the creation of a super-sensor. The result of the data aggregation may end up with errors such as false detections, misplaced object cuboids or an incorrect number of objects in the scene. The stabilization and validation process is focused on mitigating these problems. The current paper proposes four contributions for solving the stabilization and validation task, for autonomous vehicles, using the following sensors: trifocal camera, fisheye camera, long-range RADAR (Radio detection and ranging), and 4-layer and 16-layer LIDARs (Light Detection and Ranging). We propose two original data association methods used in the sensor fusion and tracking processes. The first data association algorithm is created for tracking LIDAR objects and combines multiple appearance and motion features in order to exploit the available information for road objects. The second novel data association algorithm is designed for trifocal camera objects and has the objective of finding measurement correspondences to sensor fused objects such that the super-sensor data are enriched by adding the semantic class information. The implemented trifocal object association solution uses a novel polar association scheme combined with a decision tree to find the best hypothesis–measurement correlations. Another contribution we propose for stabilizing object position and unpredictable behavior of road objects, provided by multiple types of complementary sensors, is the use of a fusion approach based on the Unscented Kalman Filter and a single-layer perceptron. The last novel contribution is related to the validation of the 3D object position, which is solved using a fuzzy logic technique combined with a semantic segmentation image. The proposed algorithms have a real-time performance, achieving a cumulative running time of 90 ms, and have been evaluated using ground truth data extracted from a high-precision GPS (global positioning system) with 2 cm accuracy, obtaining an average error of 0.8 m.

For foreign object detection on runways, highly sensitive radar sensors give the opportunity to detect even very small objects, metallic and non-metallic, also under adverse weather conditions. As it is desirable for airport applications to install only small but robust installations along the traffic areas, millimeter-wave radars offer the advantage of small antenna apertures and miniaturized system hardware. A 220-GHz radar was developed, which is capable to serve this application, if several of these are netted to cover the whole traffic area. Although under fortunate conditions the radar allows a classification or even an identification of the debris, the complete system design incorporates 3-D time-of-flight cameras for assistance in the identification process, which are also distributed along the traffic areas. The system approach further relies upon a change detection algorithm on the netted information to discriminate non-stationary alarms and reduce the false alarm ratio.

Dans cette thèse, nous étudions la robustesse d’un modèle multimodal de détection d’objets en 3D dans le contexte de véhicules autonomes. Les véhicules autonomes doivent détecter et localiser avec précision les piétons et les autres véhicules dans leur environnement 3D afin de conduire sur les routes en toute sécurité. La robustesse est l’un des aspects les plus importants d’un algorithme dans le problème de la perception 3D pour véhicules autonomes. C’est pourquoi, dans cette thèse, nous avons proposé une méthode pour évaluer la robustesse d’un modèle de détecteur d’objets en 3D. À cette fin, nous avons formé un détecteur d’objets 3D multimodal représentatif sur trois ensembles de données différents et nous avons effectué des tests sur des ensembles de données qui ont été construits avec précision pour démontrer la robustesse du modèle formé dans diverses conditions météorologiques et de luminosité. Notre méthode utilise deux approches différentes pour construire les ensembles de données proposés afin d’évaluer la robustesse. Dans une approche, nous avons utilisé des images artificiellement corrompues et dans l’autre, nous avons utilisé les images réelles dans des conditions météorologiques et de luminosité extrêmes. Afin de détecter des objets tels que des voitures et des piétons dans les scènes de circulation, le modèle multimodal s’appuie sur des images et des nuages de points 3D. Les approches multimodales pour la détection d’objets en 3D exploitent différents capteurs tels que des caméras et des détecteurs de distance pour détecter les objets d’intérêt dans l’environnement. Nous avons exploité trois ensembles de données bien connus dans le domaine de la conduite autonome, à savoir KITTI, nuScenes et Waymo. Nous avons mené des expériences approfondies pour étudier la méthode proposée afin d’évaluer la robustesse du modèle et nous avons fourni des résultats quantitatifs et qualitatifs. Nous avons observé que la méthode que nous proposons peut mesurer efficacement la robustesse du modèle.
In this thesis, we study the robustness of a multimodal 3D object detection model in the context of autonomous vehicles. Self-driving cars need to accurately detect and localize pedestrians and other vehicles in their 3D surrounding environment to drive on the roads safely. Robustness is one of the most critical aspects of an algorithm in the self-driving car 3D perception problem. Therefore, in this work, we proposed a method to evaluate a 3D object detector’s robustness. To this end, we have trained a representative multimodal 3D object detector on three different datasets. Afterward, we evaluated the trained model on datasets that we have proposed and made to assess the robustness of the trained models in diverse weather and lighting conditions. Our method uses two different approaches for building the proposed datasets for evaluating the robustness. In one approach, we used artificially corrupted images, and in the other one, we used the real images captured in diverse weather and lighting conditions. To detect objects such as cars and pedestrians in the traffic scenes, the multimodal model relies on images and 3D point clouds. Multimodal approaches for 3D object detection exploit different sensors such as camera and range detectors for detecting the objects of interest in the surrounding environment. We leveraged three well-known datasets in the domain of autonomous driving consist of KITTI, nuScenes, and Waymo. We conducted extensive experiments to investigate the proposed method for evaluating the model’s robustness and provided quantitative and qualitative results. We observed that our proposed method can measure the robustness of the model effectively.

Dans cette thèse, nous nous intéressons à la modélisation de la perception audio-visuelle avec une tête robotique. Les problèmes associés, notamment la calibration audio-visuelle, la détection, la localisation et le suivi d'objets audio-visuels sont étudiés. Une approche spatio-temporelle de calibration d'une tête robotique est proposée, basée sur une mise en correspondance probabiliste multimodale des trajectoires. Le formalisme de modèles de mélange conjugué est introduit ainsi qu'une famille d'algorithmes d'optimisation efficaces pour effectuer le regroupement multimodal. Un cas particulier de cette famille d'algorithmes, notamment l'algorithme EM conjugue, est amélioré pour obtenir des propriétés théoriques intéressantes. Des méthodes de détection d'objets multimodaux et d'estimation du nombre d'objets sont développées et leurs propriétés théoriques sont étudiées. Enfin, la méthode de regroupement multimodal proposée est combinée avec des stratégies de détection et d'estimation du nombre d'objets ainsi qu'avec des techniques de suivi pour effectuer le suivi multimodal de plusieurs objets. La performance des méthodes est démontrée sur des données simulées et réelles issues d'une base de données de scénarios audio-visuels réalistes (base de données CAVA)
In this thesis, the modelling of audio-visual perception with a head-like device is considered. The related problems, namely audio-visual calibration, audio-visual object detection, localization and tracking are addressed. A spatio-temporal approach to the head-like device calibration is proposed based on probabilistic multimodal trajectory matching. The formalism of conjugate mixture models is introduced along with a family of efficient optimization algorithms to perform multimodal clustering. One instance of this algorithm family, namely the conjugate expectation maximization (ConjEM) algorithm is further improved to gain attractive theoretical properties. The multimodal object detection and object number estimation methods are developed, their theoretical properties are discussed. Finally, the proposed multimodal clustering method is combined with the object detection and object number estimation strategies and known tracking techniques to perform multimodal multiobject tracking. The performance is demonstrated on simulated data and the database of realistic audio-visual scenarios (CAVA database)

This thesis addresses a challenging problem: “how to let the visually impaired ‘see’ others emotions”. We, human beings, are heavily dependent on facial expressions to express ourselves. A smile shows that the person you are talking to is pleased, amused, relieved etc. People use emotional information from facial expressions to switch between conversation topics and to determine attitudes of individuals. Missing emotional information from facial expressions and head gestures makes the visually impaired extremely difficult to interact with others in social events. To enhance the visually impaired’s social interactive ability, in this thesis we have been working on the scientific topic of ‘expressing human emotions through vibrotactile patterns’. It is quite challenging to deliver human emotions through touch since our touch channel is very limited. We first investigated how to render emotions through a vibrator. We developed a real time “lipless” tracking system to extract dynamic emotions from the mouth and employed mobile phones as a platform for the visually impaired to perceive primary emotion types. Later on, we extended the system to render more general dynamic media signals: for example, render live football games through vibration in the mobile for improving mobile user communication and entertainment experience. To display more natural emotions (i.e. emotion type plus emotion intensity), we developed the technology to enable the visually impaired to directly interpret human emotions. This was achieved by use of machine vision techniques and vibrotactile display. The display is comprised of a ‘vibration actuators matrix’ mounted on the back of a chair and the actuators are sequentially activated to provide dynamic emotional information. The research focus has been on finding a global, analytical, and semantic representation for facial expressions to replace state of the art facial action coding systems (FACS) approach. We proposed to use the manifold of facial expressions to characterize dynamic emotions. The basic emotional expressions with increasing intensity become curves on the manifold extended from the center. The blends of emotions lie between those curves, which could be defined analytically by the positions of the main curves. The manifold is the “Braille Code” of emotions. The developed methodology and technology has been extended for building assistive wheelchair systems to aid a specific group of disabled people, cerebral palsy or stroke patients (i.e. lacking fine motor control skills), who don’t have ability to access and control the wheelchair with conventional means, such as joystick or chin stick. The solution is to extract the manifold of the head or the tongue gestures for controlling the wheelchair. The manifold is rendered by a 2D vibration array to provide user of the wheelchair with action information from gestures and system status information, which is very important in enhancing usability of such an assistive system. Current research work not only provides a foundation stone for vibrotactile rendering system based on object localization but also a concrete step to a new dimension of human-machine interaction.
Taktil Video

abstract: Effective tactile sensing in prosthetic and robotic hands is crucial for improving the functionality of such hands and enhancing the user's experience. Thus, improving the range of tactile sensing capabilities is essential for developing versatile artificial hands. Multimodal tactile sensors called BioTacs, which include a hydrophone and a force electrode array, were used to understand how grip force, contact angle, object texture, and slip direction may be encoded in the sensor data. Findings show that slip induced under conditions of high contact angles and grip forces resulted in significant changes in both AC and DC pressure magnitude and rate of change in pressure. Slip induced under conditions of low contact angles and grip forces resulted in significant changes in the rate of change in electrode impedance. Slip in the distal direction of a precision grip caused significant changes in pressure magnitude and rate of change in pressure, while slip in the radial direction of the wrist caused significant changes in the rate of change in electrode impedance. A strong relationship was established between slip direction and the rate of change in ratios of electrode impedance for radial and ulnar slip relative to the wrist. Consequently, establishing multiple thresholds or establishing a multivariate model may be a useful method for detecting and characterizing slip. Detecting slip for low contact angles could be done by monitoring electrode data, while detecting slip for high contact angles could be done by monitoring pressure data. Predicting slip in the distal direction could be done by monitoring pressure data, while predicting slip in the radial and ulnar directions could be done by monitoring electrode data.
Dissertation/Thesis
M.S. Bioengineering 2012

The monograph reflects the problems of Russian psycholinguistics from the moment of its inception in Russia to the present day and presents its main directions that are currently developing. In addition, theoretical developments and practical results obtained in the framework of different directions and research centers are described in a concise form. The task of the book is to reflect, as far as it is possible in one edition, firstly, the history of the formation of Russian psycholinguistics; secondly, its methodology and developed methods; thirdly, the results obtained in different research centers and directions in different regions of Russia; fourthly, to outline the main directions of the further development of Russian psycholinguistics. There is no doubt that in the theoretical, methodological and applied aspects, the main problems and the results of their development by Russian psycholinguistics have no analogues in world linguistics and psycholinguistics, or are represented by completely original concepts and methods. We have tried to show this uniqueness of the problematics and the methodological equipment of Russian psycholinguistics in this book. The main role in the formation of Russian psycholinguistics was played by the Moscow psycholinguistic school of A.A. Leontyev. It still defines the main directions of Russian psycholinguistics. Russian psycholinguistics (the theory of speech activity - TSA) is based on the achievements of Russian psychology: a cultural-historical approach to the analysis of mental phenomena L.S. Vygotsky and the system-activity approach of A.N. Leontyev. Moscow is the most "psycholinguistic region" of Russia - INL RAS, Moscow State University, Moscow State Linguistic University, RUDN, Moscow State Pedagogical University, Moscow State Pedagogical University, Sechenov University, Moscow State University and other Moscow universities. Saint Petersburg psycholinguists have significant achievements, especially in the study of neurolinguistic problems, ontolinguistics. The most important feature of Russian psycholinguistics is the widespread development of psycholinguistics in the regions, the emergence of recognized psycholinguistic research centers - St. Petersburg, Tver, Saratov, Perm, Ufa, Omsk, Novosibirsk, Voronezh, Yekaterinburg, Kursk, Chelyabinsk; psycholinguistics is represented in Cherepovets, Ivanovo, Volgograd, Vyatka, Kaluga, Krasnoyarsk, Irkutsk, Vladivostok, Abakan, Maikop, Barnaul, Ulan-Ude, Yakutsk, Syktyvkar, Armavir and other cities; in Belarus - Minsk, in Ukraine - Lvov, Chernivtsi, Kharkov, in the DPR - Donetsk, in Kazakhstan - Alma-Ata, Chimkent. Our researchers work in Bulgaria, Hungary, Vietnam, China, France, Switzerland. There are Russian psycholinguists in Canada, USA, Israel, Austria and a number of other countries. All scientists from these regions and countries have contributed to the development of Russian psycholinguistics, to the development of psycholinguistic theory and methods of psycholinguistic research. Their participation has not been forgotten. We tried to present the main Russian psycholinguists in the Appendix - in the sections "Scientometrics", "Monographs and Manuals" and "Dissertations", even if there is no information about them in the Electronic Library and RSCI. The principles of including scientists in the scientometric list are presented in the Appendix. Our analysis of the content of the resulting monograph on psycholinguistic research in Russia allows us to draw preliminary conclusions about some of the distinctive features of Russian psycholinguistics: 1. cultural-historical approach to the analysis of mental phenomena of L.S.Vygotsky and the system-activity approach of A.N. Leontiev as methodological basis of Russian psycholinguistics; 2. theoretical nature of psycholinguistic research as a characteristic feature of Russian psycholinguistics. Our psycholinguistics has always built a general theory of the generation and perception of speech, mental vocabulary, linked specific research with the problems of ontogenesis, the relationship between language and thinking; 3. psycholinguistic studies of speech communication as an important subject of psycholinguistics; 4. attention to the psycholinguistic analysis of the text and the development of methods for such analysis; 5. active research into the ontogenesis of linguistic ability; 6. investigation of linguistic consciousness as one of the important subjects of psycholinguistics; 7. understanding the need to create associative dictionaries of different types as the most important practical task of psycholinguistics; 8. widespread use of psycholinguistic methods for applied purposes, active development of applied psycholinguistics. The review of the main directions of development of Russian psycholinguistics, carried out in this monograph, clearly shows that the direction associated with the study of linguistic consciousness is currently being most intensively developed in modern Russian psycholinguistics. As the practice of many years of psycholinguistic research in our country shows, the subject of study of psycholinguists is precisely linguistic consciousness - this is a part of human consciousness that is responsible for generating, understanding speech and keeping language in consciousness. Associative experiments are the core of most psycholinguistic techniques and are important both theoretically and practically. The following main areas of practical application of the results of associative experiments can be outlined. 1. Education. Associative experiments are the basis for constructing Mind Maps, one of the most promising tools for systematizing knowledge, assessing the quality, volume and nature of declarative knowledge (and using special techniques and skills). Methods based on smart maps are already widely used in teaching foreign languages, fast and deep immersion in various subject areas. 2. Information search, search optimization. The results of associative experiments can significantly improve the quality of information retrieval, its efficiency, as well as adaptability for a specific person (social group). When promoting sites (promoting them in search results), an associative experiment allows you to increase and improve the quality of the audience reached. 3. Translation studies, translation automation. An associative experiment can significantly improve the quality of translation, take into account intercultural and other social characteristics of native speakers. 4. Computational linguistics and automatic word processing. The results of associative experiments make it possible to reveal the features of a person's linguistic consciousness and contribute to the development of automatic text processing systems in a wide range of applications of natural language interfaces of computer programs and robotic solutions. 5. Advertising. The use of data on associations for specific words, slogans and texts allows you to predict and improve advertising texts. 6. Social relationships. The analysis of texts using the data of associative experiments makes it possible to assess the tonality of messages (negative / positive moods, aggression and other characteristics) based on user comments on the Internet and social networks, in the press in various projections (by individuals, events, organizations, etc.) from various social angles, to diagnose the formation of extremist ideas. 7. Content control and protection of personal data. Associative experiments improve the quality of content detection and filtering by identifying associative fields in areas subject to age restrictions, personal information, tobacco and alcohol advertising, incitement to ethnic hatred, etc. 8. Gender and individual differences. The data of associative experiments can be used to compare the reactions (and, in general, other features of thinking) between men and women, different social and age groups, representatives of different regions. The directions for the further development of Russian psycholinguistics from the standpoint of the current state of psycholinguistic science in the country are seen by us, first of all:  in the development of research in various areas of linguistic consciousness, which will contribute to the development of an important concept of speech as a verbal model of non-linguistic consciousness, in which knowledge revealed by social practice and assigned by each member of society during its inculturation is consolidated for society and on its behalf;  in the expansion of the problematics, which is formed under the influence of the growing intercultural communication in the world community, which inevitably involves the speech behavior of natural and artificial bilinguals in the new object area of psycholinguistics;  in using the capabilities of national linguistic corpora in the interests of researchers studying the functioning of non-linguistic and linguistic consciousness in speech processes;  in expanding research on the semantic perception of multimodal texts, the scope of which has greatly expanded in connection with the spread of the Internet as a means of communication in the life of modern society;  in the inclusion of the problems of professional communication and professional activity in the object area of psycholinguistics in connection with the introduction of information technologies into public practice, entailing the emergence of new professions and new features of the professional ethos;  in the further development of the theory of the mental lexicon (identifying the role of different types of knowledge in its formation and functioning, the role of the word as a unit of the mental lexicon in the formation of the image of the world, as well as the role of the natural / internal metalanguage and its specificity in speech activity);  in the broad development of associative lexicography, which will meet the most diverse needs of society and cognitive sciences. The development of associative lexicography may lead to the emergence of such disciplines as associative typology, associative variantology, associative axiology;  in expanding the spheres of applied use of psycholinguistics in social sciences, sociology, semasiology, lexicography, in the study of the brain, linguodidactics, medicine, etc. This book is a kind of summarizing result of the development of Russian psycholinguistics today. Each section provides a bibliography of studies on the relevant issue. The Appendix contains the scientometrics of leading Russian psycholinguists, basic monographs, psycholinguistic textbooks and dissertations defended in psycholinguistics. The content of the publications presented here is convincing evidence of the relevance of psycholinguistic topics and the effectiveness of the development of psycholinguistic problems in Russia.

We consider the problem of automatically extracting visual objects from web images. Despite the extraordinary advancement in deep learning, visual object detection remains a challenging task. To overcome the deficiency of pure visual techniques, we propose to make use of meta text surrounding images on the Web for enhanced detection accuracy. In this paper we present a multimodal learning algorithm to integrate text information into visual knowledge extraction. To demonstrate the effectiveness of our approach, we developed a system that takes raw webpages as input, and automatically extracts visual knowledge (e.g. object bounding boxes) from tens of millions of images crawled from the Web. Experimental results based on 46 object categories show that the extraction precision is improved significantly from 73% (with state-of-the-art deep learning programs) to 81%, which is equivalent to a 31% reduction in error rates.

Sleep staging is fundamental for sleep assessment and disease diagnosis. Although previous attempts to classify sleep stages have achieved high classification performance, several challenges remain open: 1) How to effectively extract salient waves in multimodal sleep data; 2) How to capture the multi-scale transition rules among sleep stages; 3) How to adaptively seize the key role of specific modality for sleep staging. To address these challenges, we propose SalientSleepNet, a multimodal salient wave detection network for sleep staging. Specifically, SalientSleepNet is a temporal fully convolutional network based on the $U^2$-Net architecture that is originally proposed for salient object detection in computer vision. It is mainly composed of two independent $U^2$-like streams to extract the salient features from multimodal data, respectively. Meanwhile, the multi-scale extraction module is designed to capture multi-scale transition rules among sleep stages. Besides, the multimodal attention module is proposed to adaptively capture valuable information from multimodal data for the specific sleep stage. Experiments on the two datasets demonstrate that SalientSleepNet outperforms the state-of-the-art baselines. It is worth noting that this model has the least amount of parameters compared with the existing deep neural network models.