Academic literature on the topic 'Self-attention mechanisms'
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the lists of relevant articles, books, theses, conference reports, and other scholarly sources on the topic 'Self-attention mechanisms.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Journal articles on the topic "Self-attention mechanisms"
1
Makarov, Ilya, Maria Bakhanova, Sergey Nikolenko, and Olga Gerasimova. "Self-supervised recurrent depth estimation with attention mechanisms." PeerJ Computer Science 8 (January 31, 2022): e865. http://dx.doi.org/10.7717/peerj-cs.865.
Full textAbstract:
Depth estimation has been an essential task for many computer vision applications, especially in autonomous driving, where safety is paramount. Depth can be estimated not only with traditional supervised learning but also via a self-supervised approach that relies on camera motion and does not require ground truth depth maps. Recently, major improvements have been introduced to make self-supervised depth prediction more precise. However, most existing approaches still focus on single-frame depth estimation, even in the self-supervised setting. Since most methods can operate with frame sequences, we believe that the quality of current models can be significantly improved with the help of information about previous frames. In this work, we study different ways of integrating recurrent blocks and attention mechanisms into a common self-supervised depth estimation pipeline. We propose a set of modifications that utilize temporal information from previous frames and provide new neural network architectures for monocular depth estimation in a self-supervised manner. Our experiments on the KITTI dataset show that proposed modifications can be an effective tool for exploiting temporal information in a depth prediction pipeline.
2
Bae, Ara, and Wooil Kim. "Speaker Verification Employing Combinations of Self-Attention Mechanisms." Electronics 9, no. 12 (December 21, 2020): 2201. http://dx.doi.org/10.3390/electronics9122201.
Full textAbstract:
One of the most recent speaker recognition methods that demonstrates outstanding performance in noisy environments involves extracting the speaker embedding using attention mechanism instead of average or statistics pooling. In the attention method, the speaker recognition performance is improved by employing multiple heads rather than a single head. In this paper, we propose advanced methods to extract a new embedding by compensating for the disadvantages of the single-head and multi-head attention methods. The combination method comprising single-head and split-based multi-head attentions shows a 5.39% Equal Error Rate (EER). When the single-head and projection-based multi-head attention methods are combined, the speaker recognition performance improves by 4.45%, which is the best performance in this work. Our experimental results demonstrate that the attention mechanism reflects the speaker’s properties more effectively than average or statistics pooling, and the speaker verification system could be further improved by employing combinations of different attention techniques.
3
Cao, Fude, Chunguang Zheng, Limin Huang, Aihua Wang, Jiong Zhang, Feng Zhou, Haoxue Ju, Haitao Guo, and Yuxia Du. "Research of Self-Attention in Image Segmentation." Journal of Information Technology Research 15, no. 1 (January 2022): 1–12. http://dx.doi.org/10.4018/jitr.298619.
Full textAbstract:
Although the traditional convolutional neural network is applied to image segmentation successfully, it has some limitations. That's the context information of the long-range on the image is not well captured. With the success of the introduction of self-attentional mechanisms in the field of natural language processing (NLP), people have tried to introduce the attention mechanism in the field of computer vision. It turns out that self-attention can really solve this long-range dependency problem. This paper is a summary on the application of self-attention to image segmentation in the past two years. And think about whether the self-attention module in this field can replace convolution operation in the future.
4
Dai, Biyun, Jinlong Li, and Ruoyi Xu. "Multiple Positional Self-Attention Network for Text Classification." Proceedings of the AAAI Conference on Artificial Intelligence 34, no. 05 (April 3, 2020): 7610–17. http://dx.doi.org/10.1609/aaai.v34i05.6261.
Full textAbstract:
Self-attention mechanisms have recently caused many concerns on Natural Language Processing (NLP) tasks. Relative positional information is important to self-attention mechanisms. We propose Faraway Mask focusing on the (2m + 1)-gram words and Scaled-Distance Mask putting the logarithmic distance punishment to avoid and weaken the self-attention of distant words respectively. To exploit different masks, we present Positional Self-Attention Layer for generating different Masked-Self-Attentions and a following Position-Fusion Layer in which fused positional information multiplies the Masked-Self-Attentions for generating sentence embeddings. To evaluate our sentence embeddings approach Multiple Positional Self-Attention Network (MPSAN), we perform the comparison experiments on sentiment analysis, semantic relatedness and sentence classification tasks. The result shows that our MPSAN outperforms state-of-the-art methods on five datasets and the test accuracy is improved by 0.81%, 0.6% on SST, CR datasets, respectively. In addition, we reduce training parameters and improve the time efficiency of MPSAN by lowering the dimension number of self-attention and simplifying fusion mechanism.
5
Xie, Fei, Dalong Zhang, and Chengming Liu. "Global–Local Self-Attention Based Transformer for Speaker Verification." Applied Sciences 12, no. 19 (October 10, 2022): 10154. http://dx.doi.org/10.3390/app121910154.
Full textAbstract:
Transformer models are now widely used for speech processing tasks due to their powerful sequence modeling capabilities. Previous work determined an efficient way to model speaker embeddings using the Transformer model by combining transformers with convolutional networks. However, traditional global self-attention mechanisms lack the ability to capture local information. To alleviate these problems, we proposed a novel global–local self-attention mechanism. Instead of using local or global multi-head attention alone, this method performs local and global attention in parallel in two parallel groups to enhance local modeling and reduce computational cost. To better handle local location information, we introduced locally enhanced location encoding in the speaker verification task. The experimental results of the VoxCeleb1 test set and the VoxCeleb2 dev set demonstrated the improved effect of our proposed global–local self-attention mechanism. Compared with the Transformer-based Robust Embedding Extractor Baseline System, the proposed speaker Transformer network exhibited better performance in the speaker verification task.
6
Ishizuka, Ryoto, Ryo Nishikimi, and Kazuyoshi Yoshii. "Global Structure-Aware Drum Transcription Based on Self-Attention Mechanisms." Signals 2, no. 3 (August 13, 2021): 508–26. http://dx.doi.org/10.3390/signals2030031.
Full textAbstract:
This paper describes an automatic drum transcription (ADT) method that directly estimates a tatum-level drum score from a music signal in contrast to most conventional ADT methods that estimate the frame-level onset probabilities of drums. To estimate a tatum-level score, we propose a deep transcription model that consists of a frame-level encoder for extracting the latent features from a music signal and a tatum-level decoder for estimating a drum score from the latent features pooled at the tatum level. To capture the global repetitive structure of drum scores, which is difficult to learn with a recurrent neural network (RNN), we introduce a self-attention mechanism with tatum-synchronous positional encoding into the decoder. To mitigate the difficulty of training the self-attention-based model from an insufficient amount of paired data and to improve the musical naturalness of the estimated scores, we propose a regularized training method that uses a global structure-aware masked language (score) model with a self-attention mechanism pretrained from an extensive collection of drum scores. The experimental results showed that the proposed regularized model outperformed the conventional RNN-based model in terms of the tatum-level error rate and the frame-level F-measure, even when only a limited amount of paired data was available so that the non-regularized model underperformed the RNN-based model.
7
Zhu, Hu, Ze Wang, Yu Shi, Yingying Hua, Guoxia Xu, and Lizhen Deng. "Multimodal Fusion Method Based on Self-Attention Mechanism." Wireless Communications and Mobile Computing 2020 (September 23, 2020): 1–8. http://dx.doi.org/10.1155/2020/8843186.
Full textAbstract:
Multimodal fusion is one of the popular research directions of multimodal research, and it is also an emerging research field of artificial intelligence. Multimodal fusion is aimed at taking advantage of the complementarity of heterogeneous data and providing reliable classification for the model. Multimodal data fusion is to transform data from multiple single-mode representations to a compact multimodal representation. In previous multimodal data fusion studies, most of the research in this field used multimodal representations of tensors. As the input is converted into a tensor, the dimensions and computational complexity increase exponentially. In this paper, we propose a low-rank tensor multimodal fusion method with an attention mechanism, which improves efficiency and reduces computational complexity. We evaluate our model through three multimodal fusion tasks, which are based on a public data set: CMU-MOSI, IEMOCAP, and POM. Our model achieves a good performance while flexibly capturing the global and local connections. Compared with other multimodal fusions represented by tensors, experiments show that our model can achieve better results steadily under a series of attention mechanisms.
8
POSNER, MICHAEL I., and MARY K. ROTHBART. "Developing mechanisms of self-regulation." Development and Psychopathology 12, no. 3 (September 2000): 427–41. http://dx.doi.org/10.1017/s0954579400003096.
Full textAbstract:
Child development involves both reactive and self-regulatory mechanisms that children develop in conjunction with social norms. A half-century of research has uncovered aspects of the physical basis of attentional networks that produce regulation, and has given us some knowledge of how the social environment may alter them. In this paper, we discuss six forms of developmental plasticity related to aspects of attention. We then focus on effortful or executive aspects of attention, reviewing research on temperamental individual differences and important pathways to normal and pathological development. Pathologies of development may arise when regulatory and reactive systems fail to reach the balance that allows for both self-expression and socially acceptable behavior. It remains a challenge for our society during the next millennium to obtain the information necessary to design systems that allow a successful balance to be realized by the largest possible number of children.
9
Tiwari, Prayag, Amit Kumar Jaiswal, Sahil Garg, and Ilsun You. "SANTM: Efficient Self-attention-driven Network for Text Matching." ACM Transactions on Internet Technology 22, no. 3 (August 31, 2022): 1–21. http://dx.doi.org/10.1145/3426971.
Full textAbstract:
Self-attention mechanisms have recently been embraced for a broad range of text-matching applications. Self-attention model takes only one sentence as an input with no extra information, i.e., one can utilize the final hidden state or pooling. However, text-matching problems can be interpreted either in symmetrical or asymmetrical scopes. For instance, paraphrase detection is an asymmetrical task, while textual entailment classification and question-answer matching are considered asymmetrical tasks. In this article, we leverage attractive properties of self-attention mechanism and proposes an attention-based network that incorporates three key components for inter-sequence attention: global pointwise features, preceding attentive features, and contextual features while updating the rest of the components. Our model follows evaluation on two benchmark datasets cover tasks of textual entailment and question-answer matching. The proposed efficient Self-attention-driven Network for Text Matching outperforms the state of the art on the Stanford Natural Language Inference and WikiQA datasets with much fewer parameters.
10
Ng, Hu, Glenn Jun Weng Chia, Timothy Tzen Vun Yap, and Vik Tor Goh. "Modelling sentiments based on objectivity and subjectivity with self-attention mechanisms." F1000Research 10 (May 17, 2022): 1001. http://dx.doi.org/10.12688/f1000research.73131.2.
Full textAbstract:
Background: The proliferation of digital commerce has allowed merchants to reach out to a wider customer base, prompting a study of customer reviews to gauge service and product quality through sentiment analysis. Sentiment analysis can be enhanced through subjectivity and objectivity classification with attention mechanisms. Methods: This research includes input corpora of contrasting levels of subjectivity and objectivity from different databases to perform sentiment analysis on user reviews, incorporating attention mechanisms at the aspect level. Three large corpora are chosen as the subjectivity and objectivity datasets, the Shopee user review dataset (ShopeeRD) for subjectivity, together with the Wikipedia English dataset (Wiki-en) and Internet Movie Database (IMDb) for objectivity. Word embeddings are created using Word2Vec with Skip-Gram. Then, a bidirectional LSTM with an attention layer (LSTM-ATT) imposed on word vectors. The performance of the model is evaluated and benchmarked against classification models of Logistics Regression (LR) and Linear SVC (L-SVC). Three models are trained with subjectivity (70% of ShopeeRD) and the objectivity (Wiki-en) embeddings, with ten-fold cross-validation. Next, the three models are evaluated against two datasets (IMDb and 20% of ShopeeRD). The experiments are based on benchmark comparisons, embedding comparison and model comparison with 70-10-20 train-validation-test splits. Data augmentation using AUG-BERT is performed and selected models incorporating AUG-BERT, are compared. Results: L-SVC scored the highest accuracy with 56.9% for objective embeddings (Wiki-en) while the LSTM-ATT scored 69.0% on subjective embeddings (ShopeeRD). Improved performances were observed with data augmentation using AUG-BERT, where the LSTM-ATT+AUG-BERT model scored the highest accuracy at 60.0% for objective embeddings and 70.0% for subjective embeddings, compared to 57% (objective) and 69% (subjective) for L-SVC+AUG-BERT, and 56% (objective) and 68% (subjective) for L-SVC. Conclusions: Utilizing attention layers with subjectivity and objectivity notions has shown improvement to the accuracy of sentiment analysis models.
Dissertations / Theses on the topic "Self-attention mechanisms"
1
Moadab, Ida. "The Role of Mindfulness and Self-Compassion in the Neural Mechanisms of Attention and Self-Monitoring." Thesis, University of Oregon, 2013. http://hdl.handle.net/1794/13402.
Full textAbstract:
The present study sought to investigate the effects of meditation practice on the neural mechanisms of attention and self-monitoring by comparing a group of experienced meditators to matched controls. Self-report measures of mindfulness and self-compassion were assessed to examine whether meditation-related improvements in attention and self-monitoring were linked to increases in these qualities. Thus, differences between groups (meditator versus control) on all variables and relationships among variables (attention, self-monitoring, self-compassion, and mindfulness) were explored. Results indicate that individuals with meditation experience showed enhancement in neural networks related to selective attention and attentional allocation, as evidenced by larger P1/N1 and P3b amplitudes, relative to controls. Meditators also showed improved self-monitoring of their errors, as indexed by enhanced Pe amplitudes, when compared to controls. Importantly, greater number of years of meditation experience was linked to larger Pe amplitudes, providing evidence that more practice with meditation was associated with greater error awareness. At the same time, meditators showed greater levels of mindfulness and self-compassion when compared to controls. Importantly, each of the neural indices was linked to greater levels of mindfulness and self-compassion. Specifically, self-kindness was correlated with each of these ERP components and to percentage of alpha power during meditation, and the mindfulness facet of observing fully mediated the relationship between meditation experience and P1 amplitudes. These findings suggest that the qualities that are enhanced with meditation are associated with enhancements in attentional control and awareness of errors. This study is an exciting step toward future intervention studies that combine multiple sources of information (self-report, neural measures, and behavior) to clarify the nature of the associations among these variables so that the mechanisms of mindfulness can be more fully understood.
2
Raykos, Bronwyn C. "Attentional and interpretive biases : independent dimensions of individual difference or expressions of a common selective processing mechanism?" University of Western Australia. School of Psychology, 2007. http://theses.library.uwa.edu.au/adt-WU2007.0018.
Full textAbstract:
[Truncated abstract] Attentional and interpretive biases are important dimensions of individual difference that have been implicated in the etiology and maintenance of a range of clinical problems. Yet there has been no systematic investigation into the relationship between these dimensions of individual difference. The current research program tested predictions derived from two competing theoretical accounts of the relationship between attentional and interpretive biases. The Common Mechanism Account proposes that cognitive biases represent concurrent manifestations of a single underlying selective processing mechanism. The Independent Mechanism account proposes that independent mechanisms underlie each bias. . . An apparent contradiction is that the manipulation of one bias served to also modify the other bias, despite the observation that the magnitude of the resulting change in both biases was uncorrelated. Neither the Common Mechanism nor the Independent Pathways accounts can adequately explain this pattern of results. A new account is proposed, in which attentional and interpretive biases are viewed as representing mechanisms that are related but that are not the same. Theoretical and applied implications of these findings are discussed, including the possibility that the two biases each may best predict emotional reactions to quite different stressful events and that training programs designed to attenuate allocation of attentional resources to threat may serve to reduce both attentional and interpretive selectivity in emotionally vulnerable individuals.
3
Raykos, Bronwyn C. "Attentional and interpretive biases : independent dimensions of individual difference or expressions of a common selective processing mechanism? /." Connect to this title, 2006. http://theses.library.uwa.edu.au/adt-WU2007.0018.
Full text
4
Bakkali, Souhail. "Multimodal Document Understanding with Unified Vision and Language Cross-Modal Learning." Electronic Thesis or Diss., La Rochelle, 2022. http://www.theses.fr/2022LAROS046.
Full textAbstract:
Les modèles développés dans cette thèse sont le résultat d'un processus itératif d'analyse et de synthèse entre les théories existantes et nos études réalisées. Plus spécifiquement, nous souhaitons étudier l'apprentissage inter-modal pour la compréhension contextualisée sur les composants des documents à travers le langage et la vision. Cette thèse porte sur l'avancement de la recherche sur l'apprentissage inter-modal et apporte des contributions sur quatre fronts : (i) proposer une approche inter-modale avec des réseaux profonds pour exploiter conjointement les informations visuelles et textuelles dans un espace de représentation sémantique commun afin d'effectuer et de créer automatiquement des prédictions sur les documents multimodaux; (ii) à étudier des stratégies concurrentielles pour s'attaquer aux tâches de classification de documents intermodaux, de récupération basée sur le contenu et de classification few-shot de documents ; (iii) pour résoudre les problèmes liés aux données comme l'apprentissage lorsque les données ne sont pas annotées, en proposant un réseau qui apprend des représentations génériques à partir d'une collection de documents non étiquetés ; et (iv) à exploiter les paramètres d'apprentissage few-shot lorsque les données ne contiennent que peu d’exemplesThe frameworks developed in this thesis were the outcome of an iterative process of analysis and synthesis between existing theories and our performed studies. More specifically, we wish to study cross-modality learning for contextualized comprehension on document components across language and vision. The main idea is to leverage multimodal information from document images into a common semantic space. This thesis focuses on advancing the research on cross-modality learning and makes contributions on four fronts: (i) to proposing a cross-modal approach with deep networks to jointly leverage visual and textual information into a common semantic representation space to automatically perform and make predictions about multimodal documents (i.e., the subject matter they are about); (ii) to investigating competitive strategies to address the tasks of cross-modal document classification, content-based retrieval and few-shot document classification; (iii) to addressing data-related issues like learning when data is not annotated, by proposing a network that learns generic representations from a collection of unlabeled documents; and (iv) to exploiting few-shot learning settings when data contains only few examples
5
Govindarajan, Hariprasath. "Self-Supervised Representation Learning for Content Based Image Retrieval." Thesis, Linköpings universitet, Statistik och maskininlärning, 2020. http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-166223.
Full textAbstract:
Automotive technologies and fully autonomous driving have seen a tremendous growth in recent times and have benefitted from extensive deep learning research. State-of-the-art deep learning methods are largely supervised and require labelled data for training. However, the annotation process for image data is time-consuming and costly in terms of human efforts. It is of interest to find informative samples for labelling by Content Based Image Retrieval (CBIR). Generally, a CBIR method takes a query image as input and returns a set of images that are semantically similar to the query image. The image retrieval is achieved by transforming images to feature representations in a latent space, where it is possible to reason about image similarity in terms of image content. In this thesis, a self-supervised method is developed to learn feature representations of road scenes images. The self-supervised method learns feature representations for images by adapting intermediate convolutional features from an existing deep Convolutional Neural Network (CNN). A contrastive approach based on Noise Contrastive Estimation (NCE) is used to train the feature learning model. For complex images like road scenes where mutiple image aspects can occur simultaneously, it is important to embed all the salient image aspects in the feature representation. To achieve this, the output feature representation is obtained as an ensemble of feature embeddings which are learned by focusing on different image aspects. An attention mechanism is incorporated to encourage each ensemble member to focus on different image aspects. For comparison, a self-supervised model without attention is considered and a simple dimensionality reduction approach using SVD is treated as the baseline. The methods are evaluated on nine different evaluation datasets using CBIR performance metrics. The datasets correspond to different image aspects and concern the images at different spatial levels - global, semi-global and local. The feature representations learned by self-supervised methods are shown to perform better than the SVD approach. Taking into account that no labelled data is required for training, learning representations for road scenes images using self-supervised methods appear to be a promising direction. Usage of multiple query images to emphasize a query intention is investigated and a clear improvement in CBIR performance is observed. It is inconclusive whether the addition of an attentive mechanism impacts CBIR performance. The attention method shows some positive signs based on qualitative analysis and also performs better than other methods for one of the evaluation datasets containing a local aspect. This method for learning feature representations is promising but requires further research involving more diverse and complex image aspects.
6
Sarzedas, João Pereira. "The impact of hallucination predisposition on vocal perception mechanisms : an ERP study of attention, identity and emotion." Master's thesis, 2020. http://hdl.handle.net/10451/45746.
Full textAbstract:
Tese de mestrado, Psicopatologia, Universidade de Lisboa, Faculdade de Medicina, 2020In recent years, a more dimensional view of psychotic disorders has been supported by several epidemiological studies showing the existence of a continuum of psychotic like-experiences in the general population. Reported similarities between the cognitive and neural mechanisms underpinning hallucinatory experiences in psychotic and in individuals with high hallucination predisposition provide further support for this psychosis continuum model. In the auditory domain, schizophrenic patients show altered perception of vocal information with negative content, as well as of voice identity. Whether the same voice processing mechanisms are similarly altered in individuals with high hallucination predisposition remains to be clarified. In the current study, thirty nonclinical participants with different scores on the Launay-Slade Hallucination Scale (a measure of hallucination predisposition) listened to prerecorded words varying in identity (self vs. non-self-voice) and emotional quality (negative vs. neutral vs. positive), while electroencephalographic data were recorded. Two tasks with a different attentional focus (identity focus task vs. emotional focus task) were conducted. In the identity focus task, participants attended to the identity of the voice and decided whether the voice they heard was their own voice, the voice of another person or if they were unsure. In the emotional focus task, they evaluated the emotional quality of the words. The N1, P2 and late positive potential (LPP) components were analyzed. We found a larger N1 in response to self-voice in the identity focus task and a larger LPP for the self-voice, irrespective of the attention focus. Furthermore, in the emotional focus task, we observed an association between high hallucination predisposition and a trend for a reduced LPP amplitude in response to the self-voice. The LPP result suggests the existence of an altered perception of voice identity, dependent of the attention focus, in individuals with high hallucination predisposition. Overall, the current findings partially support the psychosis continuum model.
Nas últimas décadas, uma abordagem mais dimensional das perturbações psicóticas tem ganho força com base em vários estudos epidemiológicos que demostram a existência de um continuum de experiências psicóticas na população em geral. O modelo do continuum de psicose advoga a existência de vários fenótipos de psicose num continuum que engloba indivíduos com um diagnóstico psiquiátrico (e.g., esquizofrenia) mas também indivíduos sem diagnóstico clínico que manifestam experiências do tipo psicótico. O modelo do continuum de psicose é suportado por estudos que mostram semelhanças entre os mecanismos cognitivos e neuronais que estão na base das experiências alucinatórias de pacientes psicóticos e de indivíduos sem diagnóstico clínico. As alucinações (i.e., experiências percetuais que ocorrem na ausência de estimulação externa correspondente) são um sintoma característicos das perturbações psicóticas. Contudo, também são reportadas experiências alucinatórias em pacientes com outras perturbações psiquiátricas e em indivíduos sem diagnóstico que não necessitam de apoio clínico. Os indivíduos sem diagnóstico clínico que experienciam alucinações auditivas verbais breves ou outras experiências alucinatórias são descritos na literatura como indivíduos com alta predisposição alucinatória. Numa amostra de 84711 participantes foi encontrada uma taxa de prevalência de 9.6 % de experiências alucinatórias, o que demonstra que este não é um fenómeno incomum na população em geral. A maioria das experiências alucinatórias são transitórias, mas em alguns casos persistem e aumentam o risco de transição para psicose. O estudo de experiências alucinatórias em indivíduos com predisposição alucinatória pode contribuir para uma melhor compreensão e tratamento de perturbações clínicas. Existe evidência que demonstra a existência de uma associação entre alucinações auditivas verbais e défices específicos de processamento vocal. Os modelos neurocognitivos de processamento de voz postulam que a identidade, a emoção e a fala são processados por regiões corticais funcionalmente distintas. Em indivíduos com esquizofrenia têm sido consistentemente reportadas alterações no processamento vocal da identidade, da fala e da emoção. Contudo, ainda não é claro qual o impacto destas alterações de processamento vocal no desencadear de sintomas psicóticos específicos. Examinar a existência destas alterações em indivíduos com alta predisposição alucinatória pode ser um passo útil para uma melhor compreensão da influência que estes processos têm no desencadear de experiências alucinatórias. Os escassos estudos que testaram simultaneamente o processamento de identidade, discurso e emoção parecem demonstrar diferenças de processamento vocal entre indivíduos com esquizofrenia e indivíduos com alta predisposição alucinatória. Em comparação com grupos de controlo e pacientes sem alucinações auditivas verbais, os pacientes com alucinações auditivas verbais demonstraram uma maior dificuldade no reconhecimento da sua própria voz quando esta apresentava um conteúdo negativo. Por outro lado, em indivíduos com alta predisposição alucinatória, as dificuldades no reconhecimento da própria voz não parecem estar relacionadas com o conteúdo emocional da fala. Uma das técnicas que tem vindo a demonstrar a sua utilidade para a investigação de mecanismos de perceção vocal é a técnica de potenciais evocados por eventos (event related potentials – ERP), caracterizada por elevada resolução temporal. Um dos estudos que investigou o processamento vocal da identidade, discurso e emoção com ERP encontrou efeitos de interação entre identidade (voz própria vs. voz de um desconhecido) e emoção (palavras negativas vs. neutras vs. positivas) em três componentes: N1, P2 e late positive potential (LPP). O N1 é um componente associado ao processamento sensorial da voz. O P2 é um componente associado a uma deteção precoce de saliência emocional de um estímulo. O LPP está associado a processos de atenção sustentada e de avaliação do conteúdo emocional da voz. Um estudo subsequente, utilizando um paradigma idêntico, encontrou alterações de processamento vocal, nos componentes P2 e LPP, em indivíduos com esquizofrenia. Nestes dois estudos, o conteúdo emocional foi processado implicitamente, uma vez que os participantes foram instruídos a focarem a sua atenção na identidade da voz e avaliarem se as gravações que ouviam pertenciam a eles próprios ou a outra pessoa. Alguns estudos demonstram que alterar o foco atencional de uma tarefa influencia o modo como os estímulos são processados neurofisiologicamente. Deste modo, é possível que sejam encontrados resultados diferentes numa condição onde os participantes são instruídos a focarem a sua atenção no conteúdo emocional do estímulo (i.e., avaliarem se o estímulo é negativo, neutro ou positivo). Tanto em indivíduos com esquizofrenia como em indivíduos com alta predisposição alucinatória, têm vindo a ser reportadas alterações de processos atencionais. Por conseguinte, é possível que o foco atencional module o presumível impacto da predisposição alucinatória em mecanismos de perceção de voz. No presente estudo foi explorado, através de ERPs: 1) o impacto do foco atencional na interação entre o processamento vocal da identidade e da emoção; 2) o impacto da predisposição alucinatória nestes mecanismos (foco atencional, identidade e emoção). Para esse propósito, foram realizadas duas tarefas. Em ambas as tarefas, os participantes ouviram gravações de uma voz (própria voz vs. voz de um desconhecido) a proferir palavras com conteúdo negativo, neutro ou positivo. Enquanto que na primeira tarefa (tarefa do foco na identidade) os participantes tinham de identificar a identidade da voz, na segunda tarefa (tarefa do foco na emoção) eram instruídos a avaliar o conteúdo emocional das palavras proferidas. Na tarefa de foco na identidade, procurámos replicar resultados de estudos anteriores que observaram efeitos de interação entre identidade e emoção e efeitos principais de identidade, bem como uma associação entre altos níveis de predisposição alucinatória e uma maior dificuldade no reconhecimento da própria voz. Ainda para esta tarefa, foi explorada a hipótese de que altos níveis de predisposição alucinatória predizessem um padrão neurofisiológico semelhante ao dos pacientes com esquizofrenia, uma observação que serviria de suporte para a hipótese do continuum de psicose. Na tarefa de foco na emoção era esperada uma maior amplitude de LPP em resposta a palavras negativas e positivas, em comparação com palavras neutras e, em termos exploratórios, era esperado uma influência do foco atencional no presumível impacto da predisposição alucinatória em mecanismos de perceção de voz. Foram recrutados 30 participantes “saudáveis”, sendo que o grau de predisposição alucinatória de cada individuo foi avaliado através da Launay-Slade Hallucination Scale (LSHS). No presente estudo não foram observadas interações entre identidade e emoção, nem efeitos principais de emoção. É possível que este resultado se deva a diferenças linguísticas existentes entre os estímulos usados neste estudo e os estímulos usados em estudos anteriores. Contudo, em comparação com uma voz desconhecida, a voz dos participantes elicitou uma maior amplitude de N1 na tarefa de foco na identidade e uma maior amplitude de LPP em ambas as tarefas. Estes resultados parecem sugerir que, em termos neurofisiológicos, a própria voz de uma pessoa é mais saliente do que a voz de um desconhecido. Na tarefa do foco na emoção foi também encontrada uma associação entre altos níveis de predisposição alucinatória e uma tendência para uma menor amplitude de LPP em reposta a gravações da própria voz. Este resultado sugere que o foco atencional modela o presumível impacto da predisposição alucinatória em mecanismos de perceção de voz. Ao contrário do observado em pacientes com esquizofrenia (menor amplitude de LPP em resposta a gravações da própria voz e de vozes desconhecidas com conteúdo negativo), o presumível défice de processamento vocal em indivíduos com predisposição alucinatória parece não depender do conteúdo emocional e parece ser menos marcado, uma vez que é observado apenas em resposta a gravações da própria voz e numa condição onde a identidade não está a ser explicitamente processada. Em suma, os resultados encontrados parecem servir de suporte parcial para o modelo de continuum de psicose.
7
Yeh, Han-Yun, and 葉瀚允. "End-to-end Pinyin to Character Language Model using Self-Attention Mechanism." Thesis, 2019. http://ndltd.ncl.edu.tw/handle/vk3zt8.
Full textAbstract:
碩士國立交通大學
電信工程研究所
107
Deep nerual network with conventional automatic speech recognition structure has achieved huge improvement. Similarly, end-to-end speech recongnition structure got close performance in these two years, but with huge amout of data and computing resources. This study attempt to focus on end-to-end language model, training an end-to-end language model by sequence labeling method and self-attention seq2seq model (Transformer) which are common method in some NLP task, with syllable sequence converted from 440 million words chinese corpus through a proposed G2P system. And the syllable to character model with transformer achieved lower character error rate than the baseline trigram model in our outside test set.
Books on the topic "Self-attention mechanisms"
1
Mironova, Svetlana. The doctrine of the financial and legal status of municipalities in the Russian Federation and its implementation. ru: INFRA-M Academic Publishing LLC., 2021. http://dx.doi.org/10.12737/1819040.
Full textAbstract:
The monograph forms the doctrine of the financial and legal status of municipalities and identifies the features of its implementation in the Russian Federation at the present stage. The structure of the financial and legal status of the municipality as a whole is comprehensively presented, its elements are defined, their detailed characteristics are given; the features of the implementation of the financial and legal status of certain types of municipalities (urban and municipal districts, municipal districts, urban and rural settlements, etc.), those of them that have a special legal status (single-industry towns, BUT, science towns) or are located on the territory with a special economic status (territories of advanced socio-economic development, the free port of Vladivostok), as well as within the framework of inter-municipal cooperation and the development of urban agglomerations are indicated; a system of guarantees for the implementation by municipalities is established its financial and legal status.
Particular attention is paid to the disclosure of the social block of the financial and legal status of municipalities, represented by the mechanism of citizens' participation in the financial activities of municipalities, including the involvement of citizens in the budget process at the municipal level and the implementation of public financial control over local finances.
For researchers, teachers, postgraduates and law students, employees of public authorities and local self-government.
2
Avhadeev, V., L. Bitkova, C. Bogolyubov, I. Bondarchuk, A. Vinokurov, E. Galinovskaya, D. Gorohov, et al. Implementation of the Law on Responsible Treatment of Animals: from the quality of norms to effective law enforcement. ru: INFRA-M Academic Publishing LLC., 2021. http://dx.doi.org/10.12737/1410760.
Full textAbstract:
The collection contains articles on the quality of the conceptual apparatus and terminology of Federal Law No. 498-FZ of December 27, 2018 "On Responsible Treatment of Animals and on Amendments to Certain Legislative Acts of the Russian Federation", the subject of its legal regulation, the effectiveness of the mechanism for its implementation laid down in the law, state supervision and public control in the field of animal treatment. The problems of organizing the activities of animal shelters without owners, protecting animals from abuse and responsibility for such offenses, directions and ways to improve Federal Law No. 498-FZ and the practice of its application are also highlighted. Attention is paid not only to modern, but also to historical, international and foreign experience of legal regulation of the considered social relations, norms-requirements, restrictions and prohibitions in the field of keeping and using animals, moral and ethical aspects of interaction between people and animals, which emphasizes the complex and interdisciplinary nature of the presented research.
The publication is addressed to lawyers-scientists and practitioners, subjects of the law of legislative initiative, employees of state authorities and local self-government bodies directly involved in the application of the norms of Federal Law No. 498-FZ, employees of various organizations engaged in the maintenance, use and protection of animals, animal rights activists, students and postgraduates of law schools, as well as a wide range of readers interested in this issue.
3
Andrichenko, Lyudmila, A. Postnikov, L. Vasil'eva, Zh Gaunova, E. Nikitina, and Inna Plyugina. Reform of the organization of public power: the main directions of implementation. ru: INFRA-M Academic Publishing LLC., 2021. http://dx.doi.org/10.12737/1839416.
Full textAbstract:
The monograph examines topical issues of reforming the organization of public power in our country in connection with the adoption in 2020 of the Law on Amendments to the Constitution of the Russian Federation. The logic of changes in the organization of public power and the directions of concretization of constitutional values, taking into account the laws of the development of the constitutional system of Russia, are revealed.
The most significant characteristics of the updated model of interaction of federal public authorities in accordance with the principle of separation of powers are identified, the trends of constitutional transformations in the spheres of federal relations and local self-government, ensuring the fulfillment by public authorities of international obligations of the Russian Federation are investigated. Particular attention is paid to the development of the legal mechanism of interaction between public authorities and civil society.
The authors of the book take into account the results of legislative support for the reform of public power in 2020-2021, a forecast assessment of the implementation of the relevant constitutional and legislative novelties is given, including taking into account the existing legal risks. Solutions are proposed to a number of legal issues of legislative regulation of public power, which can increase the efficiency of its functioning.
For researchers, teachers, students and postgraduates, deputies of representative authorities, state and municipal employees, as well as anyone interested in constitutional law issues.
4
Sazhina, Muza, Anna Kashirova, Stanislav Makarov, and Egor Osiop. The social wealth of the innovation system. ru: INFRA-M Academic Publishing LLC., 2022. http://dx.doi.org/10.12737/1875920.
Full textAbstract:
The monograph reveals the key socio-economic problems of the innovation economy: its content as a knowledge economy and its role in evolutionary development; human capital (living intelligence) as the main resource of the innovation economy. Much attention is paid to the institutional support of innovation through a system of institutions and mutually beneficial contracts. The mixed mechanism of implementation of innovative activity as a synthesis of spontaneous market self-regulation and conscious public administration is shown. The result of the "social control" of society and the state is the coordination of the actions of economic entities and the ordering of economic processes.
The most important institution of human society is the family as a strong power in the state. And the person himself with his knowledge, culture, ethics and morality is the main value of society. The main purpose of the family is to reproduce life and provide a person with everything necessary. The state as an institution manages a person's education and health, helps to change his lifestyle, strengthening humanity, ethics, morality and culture of life.
The modern global economy remains a sphere of domination of market egoism. It is the market that performs the function of morality as a person and society as a whole. In the global economy, a person is not a representative of the people, but a representative of the system, a standard way of life. And he should live in communication based on respect for each other.
It is concluded that today the main wealth of society is not material, but social wealth: the person himself with his knowledge, culture, ethics and morality is a living intellect; a family with the reproduction of life; immaterial knowledge that covers all types of work that cannot be calculated and paid, where the motive is the joy of free cooperation, free giving and community. In this "invisible economy" people mutually teach each other humanity and create a culture of joint thinking and living together. The State and society must preserve and increase the social wealth of human society.
For students and postgraduates of economic and managerial specialties, as well as for anyone interested in this problem.
5
O'Callaghan, Claire, and Muireann Irish. Candidate Mechanisms of Spontaneous Cognition as Revealed by Dementia Syndromes. Edited by Kalina Christoff and Kieran C. R. Fox. Oxford University Press, 2018. http://dx.doi.org/10.1093/oxfordhb/9780190464745.013.6.
Full textAbstract:
The capacity to engage in spontaneous self-generated thought is fundamental to the human experience, yet surprisingly little is known regarding the neurocognitive mechanisms that support this complex ability. Dementia syndromes offer a unique opportunity to study how the breakdown of large-scale functional brain networks impacts spontaneous cognition. Indeed, many of the characteristic cognitive changes in dementia reflect the breakdown of foundational processes essential for discrete aspects of self-generated thought. This chapter discusses how disease-specific alterations in memory-based/construction and mentalizing processes likely disrupt specific aspects of spontaneous, self-generated thought. In doing so, it provides a comprehensive overview of the neurocognitive architecture of spontaneous cognition, paying specific attention to how this sophisticated endeavor is compromised in dementia.
6
Ferguson, Robert, and Karen Gillock. Memory and Attention Adaptation Training. Oxford University Press, 2021. http://dx.doi.org/10.1093/med/9780197521526.001.0001.
Full textAbstract:
Memory and Attention Adaptation Training (MAAT) is a cognitive-behavioral therapy (CBT) designed to help cancer survivors self-manage and mitigate the late and long-term effects of cancer and cancer therapy on memory function. Cancer-related cognitive impairment (CRCI) is a set of mild to moderate memory and attention impairments that can have an adverse influence on quality of life. CRCI symptoms tend to present during active treatment, but for some individuals cognitive changes can persist for years. While the exact prevalence of CRCI is unknown, review of the literature estimates that nearly half of all survivors may experience some form of CRCI. Causes of CRCI are multiple and are the subject of continued research. Chemotherapy, genetic vulnerability, neurovascular damage, inflammation, and hormonal/endocrine disruption have all been identified as candidate mechanisms of persistent cognitive change. Given the multiple causal mechanisms, finding a biomedical treatment for CRCI remains elusive. MAAT was developed as a CBT to help cancer survivors make adaptive behavioral and cognitive changes to improve performance in the valued activities that CRCI hinders. MAAT consists of eight visits and has been designed for administration through telehealth technology, improving access to the survivorship care that so many cancer survivors may lack after the time and expense of cancer treatment. Survivors can use this workbook to reinforce their in-session learning and continue to build adaptive coping.
7
Ferguson, Robert, and Karen Gillock. Memory and Attention Adaptation Training. Oxford University Press, 2021. http://dx.doi.org/10.1093/med/9780197521571.001.0001.
Full textAbstract:
Memory and Attention Adaptation Training (MAAT) is a cognitive-behavioral therapy (CBT) designed to help cancer survivors self-manage and mitigate the late and long-term effects of cancer and cancer therapy on memory function. Cancer-related cognitive impairment (CRCI) is a set of mild to moderate memory and attention impairments that can have an adverse influence on quality of life. CRCI symptoms tend to present during active treatment, but for some individuals cognitive changes can persist for years. While the exact prevalence of CRCI is unknown, review of the literature estimates that nearly half of all survivors may experience some form of CRCI. Causes of CRCI are multiple and are the subject of continued research. Chemotherapy, genetic vulnerability, neurovascular damage, inflammation, and hormonal/endocrine disruption have all been identified as candidate mechanisms of persistent cognitive change. Given the multiple causal mechanisms, finding a biomedical treatment for CRCI remains elusive. MAAT was developed as a CBT to help cancer survivors make adaptive behavioral and cognitive changes to improve performance in the valued activities that CRCI hinders. MAAT consists of eight visits and has been designed for administration through telehealth technology, improving access to survivorship care that so many cancer survivors may lack after the time and expense of cancer treatment. Survivors are provided a workbook they can use to work with their clinician and to reinforce learning and adaptive coping. This clinician manual guides the clinician step by step on MAAT administration and provides background on the theoretical underpinnings of CRCI and MAAT.
8
Hasenkamp, Wendy. Catching the Wandering Mind. Edited by Kalina Christoff and Kieran C. R. Fox. Oxford University Press, 2018. http://dx.doi.org/10.1093/oxfordhb/9780190464745.013.12.
Full textAbstract:
This chapter considers a form of attention-based meditation as a novel means to gain insight into the mechanisms and phenomenology of spontaneous thought. Focused attention (FA) meditation involves keeping one’s attention on a chosen object, and repeatedly catching the mind when it strays from the object into spontaneous thought. This practice can thus be viewed as a kind of self-caught mind-wandering paradigm, which suggests it may have great utility for research on spontaneous thought. Current findings about the effects of meditation on mind-wandering and meta-awareness are reviewed, and implications for new research paradigms that leverage first-person reporting during FA meditation are discussed. Specifically, research recommendations are made that may enable customized analysis of individual episodes of mind-wandering and their neural correlates. It is hoped that combining detailed subjective reports from experienced meditators with rigorous objective physiological measures will advance the understanding of human consciousness.
9
Blackmore, Susan. Consciousness: A Very Short Introduction. Oxford University Press, 2017. http://dx.doi.org/10.1093/actrade/9780198794738.001.0001.
Full textAbstract:
Consciousness is ‘the last great mystery for science’. How can a physical brain create our experience of the world? What creates our identity? Do we really have free will? Could consciousness itself be an illusion? Exciting new developments in brain science are continuing these debates, and the field has now expanded to include biologists, neuroscientists, psychologists, and philosophers. Consciousness: A Very Short Introduction clarifies the potentially confusing arguments, and the major theories, whilst also outlining the amazing pace of neuroscience discoveries. Covering areas such as construction of self in the brain, mechanisms of attention, neural correlates of consciousness, and physiology of altered states of consciousness, it highlights the latest findings.
10
Han, Shihui. Cultural diversity. Oxford University Press, 2017. http://dx.doi.org/10.1093/acprof:oso/9780198743194.003.0001.
Full textAbstract:
Chapter 1 provides a brief overview of cultural differences in human behavior by giving examples of human behaviors in East Asian and Western societies. It reviews the concept of culture used by psychologists, anthropologists, and philosophers, introduces several dimensions of culture, and emphasizes shared beliefs and behavioral scripts as the key components of culture that influence human behavior. It also reviews cross-cultural psychological research that has revealed differences in multiple cognitive processes including perception, attention, memory, causal attribution, and self-reflection between individuals in East Asian and Western cultures. It gives an overview of cultural neuroscience studies that employ brain imaging techniques to reveal neural mechanisms underlying cultural differences in human behavior and mental processes.
Book chapters on the topic "Self-attention mechanisms"
1
van Dillen, Lotte F., and Esther K. Papies. "From Distraction to Mindfulness: Psychological and Neural Mechanisms of Attention Strategies in Self-Regulation." In Handbook of Biobehavioral Approaches to Self-Regulation, 141–54. New York, NY: Springer New York, 2014. http://dx.doi.org/10.1007/978-1-4939-1236-0_10.
Full text
2
Li, Jiacheng, Qiang Du, and Feifei Huang. "Research on Intrusion Detection Technology Based on CNN-SaLSTM." In Proceeding of 2021 International Conference on Wireless Communications, Networking and Applications, 456–68. Singapore: Springer Nature Singapore, 2022. http://dx.doi.org/10.1007/978-981-19-2456-9_47.
Full textAbstract:
AbstractAs Internet-connected application devices become more and more popular, more and more services need to be done through the network, which also leads to users paying more attention to network security performance. Due to the continuous iterative development of cyber attack means and attack scale, it is difficult to conduct passive security detection systems such as traditional intrusion detection mechanisms to conduct endless attacks. Later, intrusion detection was studied as an active defense technique to compensate for the shortcomings of traditional safety detection techniques. Active defense and response technology has also attracted the attention of researchers at home and abroad. The complex, engineering and large-scale scenarios presented by network attacks prevent the original passive intrusion detection system to meet the users’ needs for network security performance. With the continuous expansion of network scale, the continuous increase of network traffic scenarios and the rapid iteration of attack means, the performance of network intrusion detection system has put higher requirements. Therefore, we introduced the CNN, LSTM and self attention mechanisms in deep learning into invasion detection and performed experiments in the tensorflow framework, increasing the accuracy to 97.4%.
3
Hossain, Syed Md Minhaz, Anik Sen, and Kaushik Deb. "Detecting Spam SMS Using Self Attention Mechanism." In Intelligent Computing & Optimization, 175–84. Cham: Springer International Publishing, 2022. http://dx.doi.org/10.1007/978-3-031-19958-5_17.
Full text
4
Daanouni, Othmane, Bouchaib Cherradi, and Amal Tmiri. "Self-Attention Mechanism for Diabetic Retinopathy Detection." In Emerging Trends in ICT for Sustainable Development, 79–88. Cham: Springer International Publishing, 2021. http://dx.doi.org/10.1007/978-3-030-53440-0_10.
Full text
5
Zhang, Jiaxin, Yinshan Jia, and Hongfei Yu. "Application Research of YOLOv3 Incorporating Self-attention Mechanism." In Lecture Notes in Electrical Engineering, 454–61. Singapore: Springer Nature Singapore, 2022. http://dx.doi.org/10.1007/978-981-19-3927-3_45.
Full text
6
Kanno, Satoki, and Mamoru Mimura. "Detection of Malware Using Self-Attention Mechanism and Strings." In Network and System Security, 46–60. Cham: Springer Nature Switzerland, 2023. http://dx.doi.org/10.1007/978-3-031-39828-5_3.
Full text
7
Yuan, Genji, Jianbo Li, Yue Wang, and Xianglong Meng. "Incorporating Self Attention Mechanism into Semantic Segmentation for Lane Detection." In Wireless Algorithms, Systems, and Applications, 441–49. Cham: Springer Nature Switzerland, 2022. http://dx.doi.org/10.1007/978-3-031-19214-2_37.
Full text
8
Cvitanović, Ivana, and Marina Bagić Babac. "Deep Learning with Self-Attention Mechanism for Fake News Detection." In Studies in Computational Intelligence, 205–29. Cham: Springer International Publishing, 2021. http://dx.doi.org/10.1007/978-3-030-90087-8_10.
Full text
9
Zhang, Rui, Binbin Chen, and Hongbo Tang. "A Novel Sentiment Classification Architecture Based on Self-attention Mechanism." In 2020 International Conference on Data Processing Techniques and Applications for Cyber-Physical Systems, 685–92. Singapore: Springer Singapore, 2021. http://dx.doi.org/10.1007/978-981-16-1726-3_85.
Full text
10
Li, Xiang, Ning Sun, Jixin Liu, Lei Chai, and Haian Sun. "Multi-modal Scene Recognition Based on Global Self-attention Mechanism." In Advances in Natural Computation, Fuzzy Systems and Knowledge Discovery, 109–21. Cham: Springer International Publishing, 2023. http://dx.doi.org/10.1007/978-3-031-20738-9_14.
Full textConference papers on the topic "Self-attention mechanisms"
1
Xu, Yichong, Chenguang Zhu, Shuohang Wang, Siqi Sun, Hao Cheng, Xiaodong Liu, Jianfeng Gao, Pengcheng He, Michael Zeng, and Xuedong Huang. "Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention." In Thirty-First International Joint Conference on Artificial Intelligence {IJCAI-22}. California: International Joint Conferences on Artificial Intelligence Organization, 2022. http://dx.doi.org/10.24963/ijcai.2022/383.
Full textAbstract:
Most of today's AI systems focus on using self-attention mechanisms and transformer architectures on large amounts of diverse data to achieve impressive performance gains. In this paper, we propose to augment the transformer architecture with an external attention mechanism to bring external knowledge and context to bear. By integrating external information into the prediction process, we hope to reduce the need for ever-larger models and increase the democratization of AI systems. We find that the proposed external attention mechanism can significantly improve the performance of existing AI systems, allowing practitioners to easily customize foundation AI models to many diverse downstream applications. In particular, we focus on the task of Commonsense Reasoning, demonstrating that the proposed external attention mechanism can augment existing transformer models and significantly improve the model's reasoning capabilities. The proposed system, Knowledgeable External Attention for commonsense Reasoning (KEAR), reaches human parity on the open CommonsenseQA research benchmark with an accuracy of 89.4% in comparison to the human accuracy of 88.9%.
2
Han, Xu. "Biomedical Event Trigger Identification via Multiple Self-attention Mechanisms." In 2021 IEEE International Conference on Computer Science, Electronic Information Engineering and Intelligent Control Technology (CEI). IEEE, 2021. http://dx.doi.org/10.1109/cei52496.2021.9574606.
Full text
3
Wang, Yuchen, Kexin Shi, Chengzhuo Lu, Yuguo Liu, Malu Zhang, and Hong Qu. "Spatial-Temporal Self-Attention for Asynchronous Spiking Neural Networks." In Thirty-Second International Joint Conference on Artificial Intelligence {IJCAI-23}. California: International Joint Conferences on Artificial Intelligence Organization, 2023. http://dx.doi.org/10.24963/ijcai.2023/344.
Full textAbstract:
The brain-inspired spiking neural networks (SNNs) are receiving increasing attention due to their asynchronous event-driven characteristics and low power consumption. As attention mechanisms recently become an indispensable part of sequence dependence modeling, the combination of SNNs and attention mechanisms holds great potential for energy-efficient and high-performance computing paradigms. However, the existing works cannot benefit from both temporal-wise attention and the asynchronous characteristic of SNNs. To fully leverage the advantages of both SNNs and attention mechanisms, we propose an SNNs-based spatial-temporal self-attention (STSA) mechanism, which calculates the feature dependence across the time and space domains without destroying the asynchronous transmission properties of SNNs. To further improve the performance, we also propose a spatial-temporal relative position bias (STRPB) for STSA to consider the spatiotemporal position of spikes. Based on the STSA and STRPB, we construct a spatial-temporal spiking Transformer framework, named STS-Transformer, which is powerful and enables SNNs to work in an asynchronous event-driven manner. Extensive experiments are conducted on popular neuromorphic datasets and speech datasets, including DVS128 Gesture, CIFAR10-DVS, and Google Speech Commands, and our experimental results can outperform other state-of-the-art models.
4
Pedro, Rafael, and Arlindo L. Oliveira. "Assessing the Impact of Attention and Self-Attention Mechanisms on the Classification of Skin Lesions." In 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 2022. http://dx.doi.org/10.1109/ijcnn55064.2022.9892274.
Full text
5
Ohishi, Yasunori, Akisato Kimura, Takahito Kawanishi, Kunio Kashino, David Harwath, and James Glass. "Trilingual Semantic Embeddings of Visually Grounded Speech with Self-Attention Mechanisms." In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020. http://dx.doi.org/10.1109/icassp40776.2020.9053428.
Full text
6
Lian, Zheng, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, and Rongjun Li. "Conversational Emotion Recognition Using Self-Attention Mechanisms and Graph Neural Networks." In Interspeech 2020. ISCA: ISCA, 2020. http://dx.doi.org/10.21437/interspeech.2020-1703.
Full text
7
Chen, Qinyu, Congyi Sun, Zhonghai Lu, and Chang Gao. "Enabling Energy-Efficient Inference for Self-Attention Mechanisms in Neural Networks." In 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS). IEEE, 2022. http://dx.doi.org/10.1109/aicas54282.2022.9869924.
Full text
8
Shen, Tao, Tianyi Zhou, Guodong Long, Jing Jiang, Sen Wang, and Chengqi Zhang. "Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling." In Twenty-Seventh International Joint Conference on Artificial Intelligence {IJCAI-18}. California: International Joint Conferences on Artificial Intelligence Organization, 2018. http://dx.doi.org/10.24963/ijcai.2018/604.
Full textAbstract:
Many natural language processing tasks solely rely on sparse dependencies between a few tokens in a sentence. Soft attention mechanisms show promising performance in modeling local/global dependencies by soft probabilities between every two tokens, but they are not effective and efficient when applied to long sentences. By contrast, hard attention mechanisms directly select a subset of tokens but are difficult and inefficient to train due to their combinatorial nature. In this paper, we integrate both soft and hard attention into one context fusion model, "reinforced self-attention (ReSA)", for the mutual benefit of each other. In ReSA, a hard attention trims a sequence for a soft self-attention to process, while the soft attention feeds reward signals back to facilitate the training of the hard one. For this purpose, we develop a novel hard attention called "reinforced sequence sampling (RSS)", selecting tokens in parallel and trained via policy gradient. Using two RSS modules, ReSA efficiently extracts the sparse dependencies between each pair of selected tokens. We finally propose an RNN/CNN-free sentence-encoding model, "reinforced self-attention network (ReSAN)", solely based on ReSA. It achieves state-of-the-art performance on both the Stanford Natural Language Inference (SNLI) and the Sentences Involving Compositional Knowledge (SICK) datasets.
9
Wang, Yuxin, Yuanyuan Xie, Xiangmin Ji, Ziao Liu, and Xiaolong Liu. "RacPixGAN: An Enhanced Sketch-to-Face Synthesis GAN Based on Residual modules, Multi-Head Self-Attention Mechanisms, and CLIP Loss." In 2023 4th International Conference on Electronic Communication and Artificial Intelligence (ICECAI). IEEE, 2023. http://dx.doi.org/10.1109/icecai58670.2023.10176715.
Full text
10
Kuhn, Michael, Kazuko Fuchi, Giorgio Bazzan, Michael J. Durstock, James J. Joo, Gregory W. Reich, Richard A. Vaia, and Philip R. Buskohl. "Coupling of Geometric and Material Stiffening Mechanisms in Origami Design." In ASME 2016 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. American Society of Mechanical Engineers, 2016. http://dx.doi.org/10.1115/detc2016-60132.
Full textAbstract:
Origami, the ancient art of paper folding, has recently garnered attention from the scientific community for its capacity for unique 2D – 3D shape change and programmable mechanical properties. Application areas of such properties include packaging, self-assembly, shock absorption and deployable structures. Recent studies have highlighted the role of the folded geometry to regulate the mechanical response of the origami structures, such as the increased compression stiffness of origami tubes or the tunable in-plane stiffness through select inversion of bi-stable fold vertices. In addition to geometric re-enforcement, the mechanical response of an origami structure can also be programmed through spatial patterning of the individual fold line stiffnesses. However, the coupling between the geometric and material stiffening design spaces for origami structures is poorly understood and design rules are needed to guide the use of material stiffening to enhance or mitigate a geometric stiffening effect. In this computational study, a modal analysis of a corrugated fold with varying degrees of pre-fold and different sets of fold stiffness distributions is evaluated to highlight the interaction between geometric and material stiffness mechanisms.
Reports on the topic "Self-attention mechanisms"
1
Chornodon, Myroslava. FEAUTURES OF GENDER IN MODERN MASS MEDIA. Ivan Franko National University of Lviv, February 2021. http://dx.doi.org/10.30970/vjo.2021.49.11064.
Full textAbstract:
The article clarifies of gender identity stereotypes in modern media. The main gender stereotypes covered in modern mass media are analyzed and refuted. The model of gender relations in the media is reflected mainly in the stereotypical images of men and woman. The features of the use of gender concepts in modern periodicals for women and men were determined. The most frequently used derivatives of these macroconcepts were identified and analyzed in detail. It has been found that publications for women and men are full of various gender concepts that are used in different contexts. Ingeneral, theanalysisofthe concept-maximums and concept-minimum gender and their characteristics is carried out in the context of gender stereotypes that have been forme dand function in the society, system atizing the a ctual presentations. The study of the gender concept is relevant because it reveals new trends and features of modern gender images. Taking into account the special features of gender-labeled periodicals in general and the practical absence of comprehensive scientific studies of the gender concept in particular, there is a need to supplement Ukrainian science with this topic. Gender psychology, which is served by methods of various sciences, primarily sociological, pedagogical, linguistic, psychological, socio-psychological. Let us pay attention to linguistic and psycholinguistic methods in gender studies. Linguistic methods complement intelligence research tasks, associated with speech, word and text. Psycholinguistic methods used in gender psychology (semantic differential, semantic integral, semantic analysis of words and texts), aimed at studying speech messages, specific mechanisms of origin and perception, functions of speech activity in society, studying the relationship between speech messages and gender properties participants in the communication, to analyze the linguistic development in connection with the general development of the individual. Nowhere in gender practice there is the whole arsenal of psychological methods that allow you to explore psychological peculiarities of a person like observation, experiments, questionnaires, interviews, testing, modeling, etc. The methods of psychological self-diagnostics include: the gender aspect of the own socio-psychological portrait, a gender biography as a variant of the biographical method, aimed at the reconstruction of individual social experience. In the process of writing a gender autobiography, a person can understand the characteristics of his gender identity, as well as ways and means of their formation. Socio-psychological methods of studying gender include the study of socially constructed women’s and men’s roles, relationships and identities, sexual characteristics, psychological characteristics, etc. The use of gender indicators and gender approaches as a means of socio-psychological and sociological analysis broadens the subject boundaries of these disciplines and makes them the subject of study within these disciplines. And also, in the article a combination of concrete-historical, structural-typological, system-functional methods is implemented. Descriptive and comparative methods, method of typology, modeling are used. Also used is a method of content analysis for the study of gender content of modern gender-stamped journals. It was he who allowed quantitatively to identify and explore the features of the gender concept in the pages of periodicals for women and men. A combination of historical, structural-typological, system-functional methods is also implemented in the article. Descriptive and comparative methods, method of typology, modeling are used. A method of content analysis for the study of gender content of modern gender-labeled journals is also used. It allowed to identify and explore the features of the gender concept quantitatively in the periodicals for women and men. The conceptual perception and interpretation of the gender concept «woman», which is highlighted in the modern gender-labeled press in Ukraine, requires the elaboration of the polyfunctionality of gender interpretations, the comprehension of the metaphorical perception of this image and its role and purpose in society. A gendered approach to researching the gender content of contemporary periodicals for women and men. Conceptual analysis of contemporary gender-stamped publications within the gender conceptual sphere allows to identify and correlate the meta-gender and gender concepts that appear in society.