Academic literature on the topic 'Unsupervised anomaly detection'
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the lists of relevant articles, books, theses, conference reports, and other scholarly sources on the topic 'Unsupervised anomaly detection.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Journal articles on the topic "Unsupervised anomaly detection"
1
倪, 一鸣, and 松灿 陈. "Continual unsupervised anomaly detection." SCIENTIA SINICA Informationis 52, no. 1 (January 1, 2022): 75. http://dx.doi.org/10.1360/ssi-2021-0192.
Full text
2
Shi, Chengming, Bo Luo, Hongqi Li, Bin Li, Xinyong Mao, and Fangyu Peng. "Anomaly Detection via Unsupervised Learning for Tool Breakage Monitoring." International Journal of Machine Learning and Computing 6, no. 5 (October 2016): 256–59. http://dx.doi.org/10.18178/ijmlc.2016.6.5.607.
Full text
3
Farzad, Amir, and T. Aaron Gulliver. "Unsupervised log message anomaly detection." ICT Express 6, no. 3 (September 2020): 229–37. http://dx.doi.org/10.1016/j.icte.2020.06.003.
Full text
4
Goernitz, N., M. Kloft, K. Rieck, and U. Brefeld. "Toward Supervised Anomaly Detection." Journal of Artificial Intelligence Research 46 (February 20, 2013): 235–62. http://dx.doi.org/10.1613/jair.3623.
Full textAbstract:
Anomaly detection is being regarded as an unsupervised learning task as anomalies stem from adversarial or unlikely events with unknown distributions. However, the predictive performance of purely unsupervised anomaly detection often fails to match the required detection rates in many tasks and there exists a need for labeled data to guide the model generation. Our first contribution shows that classical semi-supervised approaches, originating from a supervised classifier, are inappropriate and hardly detect new and unknown anomalies. We argue that semi-supervised anomaly detection needs to ground on the unsupervised learning paradigm and devise a novel algorithm that meets this requirement. Although being intrinsically non-convex, we further show that the optimization problem has a convex equivalent under relatively mild assumptions. Additionally, we propose an active learning strategy to automatically filter candidates for labeling. In an empirical study on network intrusion detection data, we observe that the proposed learning methodology requires much less labeled data than the state-of-the-art, while achieving higher detection accuracies.
5
Almalawi, Abdulmohsen, Adil Fahad, Zahir Tari, Asif Irshad Khan, Nouf Alzahrani, Sheikh Tahir Bakhsh, Madini O. Alassafi, Abdulrahman Alshdadi, and Sana Qaiyum. "Add-On Anomaly Threshold Technique for Improving Unsupervised Intrusion Detection on SCADA Data." Electronics 9, no. 6 (June 18, 2020): 1017. http://dx.doi.org/10.3390/electronics9061017.
Full textAbstract:
Supervisory control and data acquisition (SCADA) systems monitor and supervise our daily infrastructure systems and industrial processes. Hence, the security of the information systems of critical infrastructures cannot be overstated. The effectiveness of unsupervised anomaly detection approaches is sensitive to parameter choices, especially when the boundaries between normal and abnormal behaviours are not clearly distinguishable. Therefore, the current approach in detecting anomaly for SCADA is based on the assumptions by which anomalies are defined; these assumptions are controlled by a parameter choice. This paper proposes an add-on anomaly threshold technique to identify the observations whose anomaly scores are extreme and significantly deviate from others, and then such observations are assumed to be ”abnormal”. The observations whose anomaly scores are significantly distant from ”abnormal” ones will be assumed as ”normal”. Then, the ensemble-based supervised learning is proposed to find a global and efficient anomaly threshold using the information of both ”normal”/”abnormal” behaviours. The proposed technique can be used for any unsupervised anomaly detection approach to mitigate the sensitivity of such parameters and improve the performance of the SCADA unsupervised anomaly detection approaches. Experimental results confirm that the proposed technique achieved a significant improvement compared to the state-of-the-art of two unsupervised anomaly detection algorithms.
6
Tian, Yu, Haihua Liao, Jing Xu, Ya Wang, Shuai Yuan, and Naijin Liu. "Unsupervised Spectrum Anomaly Detection Method for Unauthorized Bands." Space: Science & Technology 2022 (February 21, 2022): 1–10. http://dx.doi.org/10.34133/2022/9865016.
Full textAbstract:
With the rapid development of wireless communication, spectrum plays increasingly important role in both military and civilian fields. Spectrum anomaly detection aims at detecting emerging anomaly signals and spectrum usage behavior in the environment, which is indispensable to secure safety and improve spectrum efficiency. However, spectrum anomaly detection faces many difficulties, especially for unauthorized frequency bands. In unauthorized bands, the composition of spectrum is complex and the anomaly usage patterns are unknown in prior. In this paper, a Variational Autoencoder- (VAE-) based method is proposed for spectrum anomaly detection in unauthorized bands. First of all, we theoretically prove that the anomalies in unauthorized bands will introduce Background Noise Enhancement (BNE) effect and Anomaly Signal Disappearance (ASD) effects after VAE reconstruction. Then, we introduce a novel anomaly metric termed as percentile (PER) score, which focuses on capturing the distribution variation of reconstruction error caused by ASD and BNE. In order to verify the effectiveness of our method, we developed an ISM Anomaly Detection (IAD) dataset. The proposed PER score achieves superior performance against different type of anomalies. For QPSK type anomaly, our method increases the recall rate from 80% to 93% while keeping a false alarm rate of 5%. The proposed method is beneficial to broadband spectrum sensing and massive spectrum data processing. The code will be released at :QXSLAB/vae_ism_ano.git.
7
Lok, Lai Kai, Vazeerudeen Abdul Hameed, and Muhammad Ehsan Rana. "Hybrid machine learning approach for anomaly detection." Indonesian Journal of Electrical Engineering and Computer Science 27, no. 2 (August 1, 2022): 1016. http://dx.doi.org/10.11591/ijeecs.v27.i2.pp1016-1024.
Full textAbstract:
This research aims to <span lang="EN-US">improve anomaly detection performance by developing two variants of hybrid models combining supervised and unsupervised machine learning techniques. Supervised models cannot detect new or unseen types of anomaly. Hence in variant 1, a supervised model that detects normal samples is followed by an unsupervised learning model to screen anomaly. The unsupervised model is weak in differentiating between noise and fraud. Hence in variant 2, the hybrid model incorporates an unsupervised model that detects anomaly is followed by a supervised model to validate an anomaly. Three different datasets are used for model evaluation. The experiment is begun with 5 supervised models and 3 unsupervised models. After performance evaluation, 2 supervised models with the highest F1-Score and one unsupervised model with the best recall value are selected for hybrid model development. The variant 1 hybrid model recorded the best recall value across all the experiments, indicating that it is the best at detecting actual fraud and less likely to miss it compared to other models. The variant 2 hybrid model can improve the precision score significantly compared to the original unsupervised model, indicating that it is better in separating noise from fraud,</span>
8
Goldstein, Markus. "Special Issue on Unsupervised Anomaly Detection." Applied Sciences 13, no. 10 (May 11, 2023): 5916. http://dx.doi.org/10.3390/app13105916.
Full text
9
Zhou, Wei, Yuan Gao, Jianhang Ji, Shicheng Li, and Yugen Yi. "Unsupervised Anomaly Detection for Glaucoma Diagnosis." Wireless Communications and Mobile Computing 2021 (October 1, 2021): 1–14. http://dx.doi.org/10.1155/2021/5978495.
Full textAbstract:
With the rapid development of high tech, Internet of Things (IoT) and artificial intelligence (AI) achieve a series of achievements in the healthcare industry. Among them, automatic glaucoma diagnosis is one of them. Glaucoma is second leading cause of blindness in the world. Although many automatic glaucoma diagnosis approaches have been proposed, they still face the following two challenges. First, the data acquisition of diseased images is extremely expensive, especially for disease with low occurrence, leading to the class imbalance. Second, large-scale labeled data are hard to obtain in medical image domain. The aforementioned challenges limit the practical application of these approaches in glaucoma diagnosis. To address these disadvantages, this paper proposes an unsupervised anomaly detection framework based on sparse principal component analysis (SPCA) for glaucoma diagnosis. In the proposed approach, we just employ the one-class normal (nonglaucoma) images for training, so the class imbalance problem can be avoided. Then, to distinguish the glaucoma (abnormal) images from the normal images, a feature set consisting of segmentation-based features and image-based features is extracted, which can capture the shape and textural changes. Next, SPCA is adopted to select the effective features from the feature set. Finally, with the usage of the extracted effective features, glaucoma diagnosis can be automatically accomplished via introducing the T 2 statistic and the control limit, overcoming the issue of insufficient labeled samples. Extensive experiments are carried out on the two public databases, and the experimental results verify the effectiveness of the proposed approach.
10
Chung, Hwehee, Jongho Park, Jongsoo Keum, Hongdo Ki, and Seokho Kang. "Unsupervised Anomaly Detection Using Style Distillation." IEEE Access 8 (2020): 221494–502. http://dx.doi.org/10.1109/access.2020.3043473.
Full textDissertations / Theses on the topic "Unsupervised anomaly detection"
1
Mazel, Johan. "Unsupervised network anomaly detection." Thesis, Toulouse, INSA, 2011. http://www.theses.fr/2011ISAT0024/document.
Full textAbstract:
La détection d'anomalies est une tâche critique de l'administration des réseaux. L'apparition continue de nouvelles anomalies et la nature changeante du trafic réseau compliquent de fait la détection d'anomalies. Les méthodes existantes de détection d'anomalies s'appuient sur une connaissance préalable du trafic : soit via des signatures créées à partir d'anomalies connues, soit via un profil de normalité. Ces deux approches sont limitées : la première ne peut détecter les nouvelles anomalies et la seconde requiert une constante mise à jour de son profil de normalité. Ces deux aspects limitent de façon importante l'efficacité des méthodes de détection existantes.Nous présentons une approche non-supervisée qui permet de détecter et caractériser les anomalies réseaux de façon autonome. Notre approche utilise des techniques de partitionnement afin d'identifier les flux anormaux. Nous proposons également plusieurs techniques qui permettent de traiter les anomalies extraites pour faciliter la tâche des opérateurs. Nous évaluons les performances de notre système sur des traces de trafic réel issues de la base de trace MAWI. Les résultats obtenus mettent en évidence la possibilité de mettre en place des systèmes de détection d'anomalies autonomes et fonctionnant sans connaissance préalableAnomaly detection has become a vital component of any network in today’s Internet. Ranging from non-malicious unexpected events such as flash-crowds and failures, to network attacks such as denials-of-service and network scans, network traffic anomalies can have serious detrimental effects on the performance and integrity of the network. The continuous arising of new anomalies and attacks create a continuous challenge to cope with events that put the network integrity at risk. Moreover, the inner polymorphic nature of traffic caused, among other things, by a highly changing protocol landscape, complicates anomaly detection system's task. In fact, most network anomaly detection systems proposed so far employ knowledge-dependent techniques, using either misuse detection signature-based detection methods or anomaly detection relying on supervised-learning techniques. However, both approaches present major limitations: the former fails to detect and characterize unknown anomalies (letting the network unprotected for long periods) and the latter requires training over labeled normal traffic, which is a difficult and expensive stage that need to be updated on a regular basis to follow network traffic evolution. Such limitations impose a serious bottleneck to the previously presented problem.We introduce an unsupervised approach to detect and characterize network anomalies, without relying on signatures, statistical training, or labeled traffic, which represents a significant step towards the autonomy of networks. Unsupervised detection is accomplished by means of robust data-clustering techniques, combining Sub-Space clustering with Evidence Accumulation or Inter-Clustering Results Association, to blindly identify anomalies in traffic flows. Correlating the results of several unsupervised detections is also performed to improve detection robustness. The correlation results are further used along other anomaly characteristics to build an anomaly hierarchy in terms of dangerousness. Characterization is then achieved by building efficient filtering rules to describe a detected anomaly. The detection and characterization performances and sensitivities to parameters are evaluated over a substantial subset of the MAWI repository which contains real network traffic traces.Our work shows that unsupervised learning techniques allow anomaly detection systems to isolate anomalous traffic without any previous knowledge. We think that this contribution constitutes a great step towards autonomous network anomaly detection.This PhD thesis has been funded through the ECODE project by the European Commission under the Framework Programme 7. The goal of this project is to develop, implement, and validate experimentally a cognitive routing system that meet the challenges experienced by the Internet in terms of manageability and security, availability and accountability, as well as routing system scalability and quality. The concerned use case inside the ECODE project is network anomaly
2
Joshi, Vineet. "Unsupervised Anomaly Detection in Numerical Datasets." University of Cincinnati / OhioLINK, 2015. http://rave.ohiolink.edu/etdc/view?acc_num=ucin1427799744.
Full text
3
Di, Felice Marco. "Unsupervised anomaly detection in HPC systems." Master's thesis, Alma Mater Studiorum - Università di Bologna, 2019.
Find full textAbstract:
Alla base di questo studio vi è l'analisi di tecniche non supervisionate applicate per il rilevamento di stati anomali in sistemi HPC, complessi calcolatori capaci di raggiungere prestazioni dell'ordine dei PetaFLOPS. Nel mondo HPC, per anomalia si intende un particolare stato che induce un cambiamento delle prestazioni rispetto al normale funzionamento del sistema. Le anomalie possono essere di natura diversa come il guasto che può riguardare un componente, una configurazione errata o un'applicazione che entra in uno stato inatteso provocando una prematura interruzione dei processi. I datasets utilizzati in un questo progetto sono stati raccolti da D.A.V.I.D.E., un reale sistema HPC situato presso il CINECA di Casalecchio di Reno, o sono stati generati simulando lo stato di un singolo nodo di un virtuale sistema HPC analogo a quello del CINECA modellato secondo specifiche funzioni non lineari ma privo di rumore. Questo studio propone un approccio inedito, quello non supervisionato, mai applicato prima per svolgere anomaly detection in sistemi HPC. Si è focalizzato sull'individuazione dei possibili vantaggi indotti dall'uso di queste tecniche applicate in tale campo. Sono stati realizzati e mostrati alcuni casi che hanno prodotto raggruppamenti interessanti attraverso le combinazioni di Variational Autoencoders, un particolare tipo di autoencoder probabilistico con la capacità di preservare la varianza dell'input set nel suo spazio latente, e di algoritmi di clustering, come K-Means, DBSCAN, Gaussian Mixture ed altri già noti in letteratura.
4
Forstén, Andreas. "Unsupervised Anomaly Detection in Receipt Data." Thesis, KTH, Skolan för datavetenskap och kommunikation (CSC), 2017. http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-215161.
Full textAbstract:
With the progress of data handling methods and computing power comes the possibility of automating tasks that are not necessarily handled by humans. This study was done in cooperation with a company that digitalizes receipts for companies. We investigate the possibility of automating the task of finding anomalous receipt data, which could automate the work of receipt auditors. We study both anomalous user behaviour and individual receipts. The results indicate that automation is possible, which may reduce the necessity of human inspection of receipts.Med de framsteg inom datahantering och datorkraft som gjorts så kommer också möjligheten att automatisera uppgifter som ej nödvändigtvis utförs av människor. Denna studie gjordes i samarbete med ett företag som digitaliserar företags kvitton. Vi undersöker möjligheten att automatisera sökandet av avvikande kvittodata, vilket kan avlasta revisorer. Vti studerar både avvikande användarbeteenden och individuella kvitton. Resultaten indikerar att automatisering är möjligt, vilket kan reducera behovet av mänsklig inspektion av kvitton
5
Cheng, Leon. "Unsupervised topic discovery by anomaly detection." Thesis, Monterey, California: Naval Postgraduate School, 2013. http://hdl.handle.net/10945/37599.
Full textAbstract:
Approved for public release; distribution is unlimitedWith the vast amount of information and public comment available online, it is of increasing interest to understand what is being said and what topics are trending online. Government agencies, for example, want to know what policies concern the public without having to look through thousands of comments manually. Topic detection provides automatic identification of topics in documents based on the information content and enhances many natural language processing tasks, including text summarization and information retrieval. Unsupervised topic detection, however, has always been a difficult task. Methods such as Latent Dirichlet Allocation (LDA) convert documents from word space into document space (weighted sums over topic space), but do not perform any form of classification, nor do they address the relation of generated topics with actual human level topics. In this thesis we attempt a novel way of unsupervised topic detection and classification by performing LDA and then clustering. We propose variations to the popular K-Mean Clustering algorithm to optimize the choice of centroids, and we perform experiments using Facebook data and the New York Times (NYT) corpus. Although the results were poor for the Facebook data, our method performed acceptably with the NYT data. The new clustering algorithms also performed slightly and consistently better than the normal K-Means algorithm.
6
Putina, Andrian. "Unsupervised anomaly detection : methods and applications." Electronic Thesis or Diss., Institut polytechnique de Paris, 2022. http://www.theses.fr/2022IPPAT012.
Full textAbstract:
Une anomalie (également connue sous le nom de outlier) est une instance qui s'écarte de manière significative du reste des données et est définie par Hawkins comme "une observation, qui s'écarte tellement des autres observations qu'elle éveille les soupçons qu'il a été généré par un mécanisme différent". La détection d’anomalies (également connue sous le nom de détection de valeurs aberrantes ou de nouveauté) est donc le domaine de l’apprentissage automatique et de l’exploration de données dans le but d’identifier les instances dont les caractéristiques semblent être incohérentes avec le reste de l’ensemble de données. Dans de nombreuses applications, distinguer correctement l'ensemble des points de données anormaux (outliers) de l'ensemble des points normaux (inliers) s'avère très important. Une première application est le nettoyage des données, c'est-à-dire l'identification des mesures bruyantes et fallacieuses dans un ensemble de données avant d'appliquer davantage les algorithmes d'apprentissage. Cependant, avec la croissance explosive du volume de données pouvant être collectées à partir de diverses sources, par exemple les transactions par carte, les connexions Internet, les mesures de température, etc., l'utilisation de la détection d'anomalies devient une tâche autonome cruciale pour la surveillance continue des systèmes. Dans ce contexte, la détection d'anomalies peut être utilisée pour détecter des attaques d'intrusion en cours, des réseaux de capteurs défaillants ou des masses cancéreuses. La thèse propose d'abord une approche basée sur un collection d'arbres pour la détection non supervisée d'anomalies, appelée "Random Histogram Forest (RHF)". L'algorithme résout le problème de la dimensionnalité en utilisant le quatrième moment central (alias 'kurtosis') dans la construction du modèle en bénéficiant d'un temps d'exécution linéaire. Un moteur de détection d'anomalies basé sur le stream, appelé 'ODS', qui exploite DenStream, une technique de clustering non supervisée est présenté par la suite et enfin un moteur de détection automatisée d'anomalies qui allège l'effort humain requis lorsqu'il s'agit de plusieurs algorithmes et hyper-paramètres est présenté en dernière contributionAn anomaly (also known as outlier) is an instance that significantly deviates from the rest of the input data and being defined by Hawkins as 'an observation, which deviates so much from other observations as to arouse suspicions that it was generated by a different mechanism'. Anomaly detection (also known as outlier or novelty detection) is thus the machine learning and data mining field with the purpose of identifying those instances whose features appear to be inconsistent with the remainder of the dataset. In many applications, correctly distinguishing the set of anomalous data points (outliers) from the set of normal ones (inliers) proves to be very important. A first application is data cleaning, i.e., identifying noisy and fallacious measurement in a dataset before further applying learning algorithms. However, with the explosive growth of data volume collectable from various sources, e.g., card transactions, internet connections, temperature measurements, etc. the use of anomaly detection becomes a crucial stand-alone task for continuous monitoring of the systems. In this context, anomaly detection can be used to detect ongoing intrusion attacks, faulty sensor networks or cancerous masses.The thesis proposes first a batch tree-based approach for unsupervised anomaly detection, called 'Random Histogram Forest (RHF)'. The algorithm solves the curse of dimensionality problem using the fourth central moment (aka kurtosis) in the model construction while boasting linear running time. A stream based anomaly detection engine, called 'ODS', that leverages DenStream, an unsupervised clustering technique is presented subsequently and finally Automated Anomaly Detection engine which alleviates the human effort required when dealing with several algorithm and hyper-parameters is presented as last contribution
7
Audibert, Julien. "Unsupervised anomaly detection in time-series." Electronic Thesis or Diss., Sorbonne université, 2021. http://www.theses.fr/2021SORUS358.
Full textAbstract:
La détection d'anomalies dans les séries temporelles multivariées est un enjeu majeur dans de nombreux domaines. La complexité croissante des systèmes et l'explosion de la quantité de données ont rendu son automatisation indispensable. Cette thèse propose une méthode non supervisée de détection d'anomalies dans les séries temporelles multivariées appelée USAD. Cependant, les méthodes de réseaux de neurones profonds souffrent d'une limitation dans leur capacité à extraire des caractéristiques des données puisqu'elles ne s'appuient que sur des informations locales. Afin d'améliorer les performances de ces méthodes, cette thèse présente une stratégie d'ingénierie des caractéristiques qui introduit des informations non-locales. Enfin, cette thèse propose une comparaison de seize méthodes de détection d'anomalies dans les séries temporelles pour comprendre si l'explosion de la complexité des méthodes de réseaux de neurones proposées dans les publications actuelles est réellement nécessaireAnomaly detection in multivariate time series is a major issue in many fields. The increasing complexity of systems and the explosion of the amount of data have made its automation indispensable. This thesis proposes an unsupervised method for anomaly detection in multivariate time series called USAD. However, deep neural network methods suffer from a limitation in their ability to extract features from the data since they only rely on local information. To improve the performance of these methods, this thesis presents a feature engineering strategy that introduces non-local information. Finally, this thesis proposes a comparison of sixteen time series anomaly detection methods to understand whether the explosion in complexity of neural network methods proposed in the current literature is really necessary
8
Dani, Mohamed Cherif. "Unsupervised anomaly detection for aircraft health monitoring system." Thesis, Sorbonne Paris Cité, 2017. http://www.theses.fr/2017USPCB258.
Full textAbstract:
La limite des connaissances techniques ou fondamentale, est une réalité dont l’industrie fait face. Le besoin de mettre à jour cette connaissance acquise est essentiel pour une compétitivité économique, mais aussi pour une meilleure maniabilité des systèmes et machines. Aujourd’hui grâce à ces systèmes et machine, l’expansion de données en quantité, en fréquence de génération est un véritable phénomène. À présent par exemple, les avions Airbus génèrent des centaines de mégas de données par jour, et intègrent des centaines voire des milliers de capteurs dans les nouvelles générations d’avions. Ces données générées par ces capteurs, sont exploitées au sol ou pendant le vol, pour surveiller l’état et la santé de l’avion, et pour détecter des pannes, des incidents ou des changements. En théorie, ces pannes, ces incidents ou ces changements sont connus sous le terme d’anomalie. Une anomalie connue comme un comportement qui ne correspond pas au comportement normal des données. Certains la définissent comme une déviation d’un modèle normal, d’autres la définissent comme un changement. Quelques soit la définition, le besoin de détecter cette anomalie est important pour le bon fonctionnement de l'avion. Actuellement, la détection des anomalies à bord des avions est assuré par plusieurs équipements de surveillance aéronautiques, l’un de ces équipements est le « Aircraft condition monitoring System –ACMS », enregistre les données générées par les capteurs en continu, il surveille aussi l’avion en temps réel grâce à des triggers et des seuils programmés par des Airlines ou autres mais à partir d’une connaissance a priori du système. Cependant, plusieurs contraintes limitent le bon fonctionnement de cet équipement, on peut citer par exemple, la limitation des connaissances humaines un problème classique que nous rencontrons dans plusieurs domaines. Cela veut dire qu’un trigger ne détecte que les anomalies et les incidents dont il est désigné, et si une nouvelle condition surgit suite à une maintenance, changement de pièce, etc. Le trigger est incapable s’adapter à cette nouvelle condition, et il va labéliser toute cette nouvelle condition comme étant une anomalie. D’autres problèmes et contraintes seront cités progressivement dans les chapitres qui suivent. L’objectif principal de notre travail est de détecter les anomalies et les changements dans les données de capteurs, afin d’améliorer le system de surveillance de santé d’avion connu sous le nom Aircraft Health Monitoring(AHM). Ce travail est basé principalement sur une analyse à deux étapes, Une analyse unie varie dans un contexte non supervisé, qui nous permettra de se focaliser sur le comportement de chaque capteur indépendamment, et de détecter les différentes anomalies et changements pour chaque capteur. Puis une analyse multi-variée qui nous permettra de filtrer certaines anomalies détectées (fausses alarmes) dans la première analyse et de détecter des groupes de comportement suspects. La méthode est testée sur des données réelles et synthétiques, où les résultats, l’identification et la validation des anomalies sont discutées dans cette thèseThe limitation of the knowledge, technical, fundamental is a daily challenge for industries. The need to updates these knowledge are important for a competitive industry and also for an efficient reliability and maintainability of the systems. Actually, thanks to these machines and systems, the expansion of the data on quantity and frequency of generation is a real phenomenon. Within Airbus for example, and thanks to thousands of sensors, the aircrafts generate hundreds of megabytes of data per flight. These data are today exploited on the ground to improve safety and health monitoring system as a failure, incident and change detection. In theory, these changes, incident and failure are known as anomalies. An anomaly is known as deviation form a normal behavior of the data. Others define it as a behavior that do not conform the normal behavior. Whatever the definition, the anomaly detection process is very important for good functioning of the aircraft. Currently, the anomaly detection process is provided by several health monitoring equipments, one of these equipment is the Aircraft Health Monitoring System (ACMS), it records continuously the date of each sensor, and also monitor these sensors to detect anomalies and incident using triggers and predefined condition (exeedance approach). These predefined conditions are programmed by airlines and system designed according to a prior knowledge (physical, mechanical, etc.). However, several constraints limit the ACMS anomaly detection potential. We can mention, for example, the limitation the expert knowledge which is a classic problem in many domains, since the triggers are designed only to the targeted anomalies. Otherwise, the triggers do not cover all the system conditions. In other words, if a new behavior appears (new condition) in the sensor, after a maintenance action, parts changing, etc. the predefined conditions won't detect any thing and may be in many cases generated false alarms. Another constraint is that the triggers (predefined conditions) are static, they are unable to adapt their proprieties to each new condition. Another limitation is discussed gradually in the future chapters. The principle of objective of this thesis is to detect anomalies and changes in the ACMS data. In order to improve the health monitoring function of the ACMS. The work is based principally on two stages, the univariate anomaly detection stage, where we use the unsupervised learning to process the univariate sensors, since we don’t have any a prior knowledge of the system, and no documentation or labeled classes are available. The univariate analysis focuses on each sensor independently. The second stage of the analysis is the multivariate anomaly detection, which is based on density clustering, where the objective is to filter the anomalies detected in the first stage (false alarms) and to detect suspected behaviours (group of anomalies). The anomalies detected in both univariate and multivariate can be potential triggers or can be used to update the existing triggers. Otherwise, we propose also a generic concept of anomaly detection based on univariate and multivariate anomaly detection. And finally a new concept of validation anomalies within airbus
9
Dani, Mohamed Cherif. "Unsupervised anomaly detection for aircraft health monitoring system." Electronic Thesis or Diss., Sorbonne Paris Cité, 2017. http://www.theses.fr/2017USPCB258.
Full textAbstract:
La limite des connaissances techniques ou fondamentale, est une réalité dont l’industrie fait face. Le besoin de mettre à jour cette connaissance acquise est essentiel pour une compétitivité économique, mais aussi pour une meilleure maniabilité des systèmes et machines. Aujourd’hui grâce à ces systèmes et machine, l’expansion de données en quantité, en fréquence de génération est un véritable phénomène. À présent par exemple, les avions Airbus génèrent des centaines de mégas de données par jour, et intègrent des centaines voire des milliers de capteurs dans les nouvelles générations d’avions. Ces données générées par ces capteurs, sont exploitées au sol ou pendant le vol, pour surveiller l’état et la santé de l’avion, et pour détecter des pannes, des incidents ou des changements. En théorie, ces pannes, ces incidents ou ces changements sont connus sous le terme d’anomalie. Une anomalie connue comme un comportement qui ne correspond pas au comportement normal des données. Certains la définissent comme une déviation d’un modèle normal, d’autres la définissent comme un changement. Quelques soit la définition, le besoin de détecter cette anomalie est important pour le bon fonctionnement de l'avion. Actuellement, la détection des anomalies à bord des avions est assuré par plusieurs équipements de surveillance aéronautiques, l’un de ces équipements est le « Aircraft condition monitoring System –ACMS », enregistre les données générées par les capteurs en continu, il surveille aussi l’avion en temps réel grâce à des triggers et des seuils programmés par des Airlines ou autres mais à partir d’une connaissance a priori du système. Cependant, plusieurs contraintes limitent le bon fonctionnement de cet équipement, on peut citer par exemple, la limitation des connaissances humaines un problème classique que nous rencontrons dans plusieurs domaines. Cela veut dire qu’un trigger ne détecte que les anomalies et les incidents dont il est désigné, et si une nouvelle condition surgit suite à une maintenance, changement de pièce, etc. Le trigger est incapable s’adapter à cette nouvelle condition, et il va labéliser toute cette nouvelle condition comme étant une anomalie. D’autres problèmes et contraintes seront cités progressivement dans les chapitres qui suivent. L’objectif principal de notre travail est de détecter les anomalies et les changements dans les données de capteurs, afin d’améliorer le system de surveillance de santé d’avion connu sous le nom Aircraft Health Monitoring(AHM). Ce travail est basé principalement sur une analyse à deux étapes, Une analyse unie varie dans un contexte non supervisé, qui nous permettra de se focaliser sur le comportement de chaque capteur indépendamment, et de détecter les différentes anomalies et changements pour chaque capteur. Puis une analyse multi-variée qui nous permettra de filtrer certaines anomalies détectées (fausses alarmes) dans la première analyse et de détecter des groupes de comportement suspects. La méthode est testée sur des données réelles et synthétiques, où les résultats, l’identification et la validation des anomalies sont discutées dans cette thèseThe limitation of the knowledge, technical, fundamental is a daily challenge for industries. The need to updates these knowledge are important for a competitive industry and also for an efficient reliability and maintainability of the systems. Actually, thanks to these machines and systems, the expansion of the data on quantity and frequency of generation is a real phenomenon. Within Airbus for example, and thanks to thousands of sensors, the aircrafts generate hundreds of megabytes of data per flight. These data are today exploited on the ground to improve safety and health monitoring system as a failure, incident and change detection. In theory, these changes, incident and failure are known as anomalies. An anomaly is known as deviation form a normal behavior of the data. Others define it as a behavior that do not conform the normal behavior. Whatever the definition, the anomaly detection process is very important for good functioning of the aircraft. Currently, the anomaly detection process is provided by several health monitoring equipments, one of these equipment is the Aircraft Health Monitoring System (ACMS), it records continuously the date of each sensor, and also monitor these sensors to detect anomalies and incident using triggers and predefined condition (exeedance approach). These predefined conditions are programmed by airlines and system designed according to a prior knowledge (physical, mechanical, etc.). However, several constraints limit the ACMS anomaly detection potential. We can mention, for example, the limitation the expert knowledge which is a classic problem in many domains, since the triggers are designed only to the targeted anomalies. Otherwise, the triggers do not cover all the system conditions. In other words, if a new behavior appears (new condition) in the sensor, after a maintenance action, parts changing, etc. the predefined conditions won't detect any thing and may be in many cases generated false alarms. Another constraint is that the triggers (predefined conditions) are static, they are unable to adapt their proprieties to each new condition. Another limitation is discussed gradually in the future chapters. The principle of objective of this thesis is to detect anomalies and changes in the ACMS data. In order to improve the health monitoring function of the ACMS. The work is based principally on two stages, the univariate anomaly detection stage, where we use the unsupervised learning to process the univariate sensors, since we don’t have any a prior knowledge of the system, and no documentation or labeled classes are available. The univariate analysis focuses on each sensor independently. The second stage of the analysis is the multivariate anomaly detection, which is based on density clustering, where the objective is to filter the anomalies detected in the first stage (false alarms) and to detect suspected behaviours (group of anomalies). The anomalies detected in both univariate and multivariate can be potential triggers or can be used to update the existing triggers. Otherwise, we propose also a generic concept of anomaly detection based on univariate and multivariate anomaly detection. And finally a new concept of validation anomalies within airbus
10
Sarossy, George. "Anomaly detection in Network data with unsupervised learning methods." Thesis, Mälardalens högskola, Akademin för innovation, design och teknik, 2021. http://urn.kb.se/resolve?urn=urn:nbn:se:mdh:diva-55096.
Full textAbstract:
Anomaly detection has become a crucial part of the protection of information and integrity. Due to the increase of cyber threats the demand for anomaly detection has grown for companies. Anomaly detection on time series data aims to detect unexpected behavior on the system. Anomalies often occur online, and companies need to be able to protect themselves from these intrusions. Multiple machine learning algorithms have been used and researched to solve the problem with anomaly detection and it is ongoing research to find the most optimal algorithms. Therefore, this study investigates algorithms such as K-means, Mean Shift and DBSCAN algorithm could be a solution for the problem. The study also investigates if combining the algorithms will improve the result. The results that the study reveals that the combinations of the algorithms perform slightly worse than the individual algorithms regarding speed and accuracy to detect anomalies. The algorithms without combinations did perform well during this study, they have slight differences between each other, and the results show the DBSCAN algorithm has slightly better total detection compared to the other algorithms and has slower execution time. The conclusion for this study reveals that the Mean Shift algorithm had the fastest execution time and the DBSCAN algorithm had the highest accuracy. The study also reveals most of the combinations between the algorithms did not improve during the fusion. However, the DBSCAN + Mean Shift fusion did improve the accuracy, and the K-means + Mean Shift fusion did improve the execution time.
Book chapters on the topic "Unsupervised anomaly detection"
1
Deepak, P. "Anomaly Detection for Data with Spatial Attributes." In Unsupervised Learning Algorithms, 1–32. Cham: Springer International Publishing, 2016. http://dx.doi.org/10.1007/978-3-319-24211-8_1.
Full text
2
Angiulli, Fabrizio, Fabio Fassetti, Luca Ferragina, and Rosaria Spada. "Cooperative Deep Unsupervised Anomaly Detection." In Discovery Science, 318–28. Cham: Springer Nature Switzerland, 2022. http://dx.doi.org/10.1007/978-3-031-18840-4_23.
Full text
3
Simarro Viana, Jaime, Ezequiel de la Rosa, Thijs Vande Vyvere, David Robben, Diana M. Sima, and CENTER-TBI Participants and Investigators. "Unsupervised 3D Brain Anomaly Detection." In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, 133–42. Cham: Springer International Publishing, 2021. http://dx.doi.org/10.1007/978-3-030-72084-1_13.
Full text
4
Higuera, Juan Ramón Bermejo, Javier Bermejo Higuera, Juan Antonio Sicilia Montalvo, and Rubén González Crespo. "Unsupervised Approaches in Anomaly Detection." In Intelligent Systems Reference Library, 57–83. Cham: Springer Nature Switzerland, 2024. http://dx.doi.org/10.1007/978-3-031-54038-7_3.
Full text
5
Zhao, Zhiruo, Kishan G. Mehrotra, and Chilukuri K. Mohan. "Ensemble Algorithms for Unsupervised Anomaly Detection." In Current Approaches in Applied Artificial Intelligence, 514–25. Cham: Springer International Publishing, 2015. http://dx.doi.org/10.1007/978-3-319-19066-2_50.
Full text
6
Zimmerer, David, Daniel Paech, Carsten Lüth, Jens Petersen, Gregor Köhler, and Klaus Maier-Hein. "Unsupervised Anomaly Detection in the Wild." In Informatik aktuell, 26–31. Wiesbaden: Springer Fachmedien Wiesbaden, 2022. http://dx.doi.org/10.1007/978-3-658-36932-3_6.
Full text
7
Graß, Alexander, Christian Beecks, and Jose Angel Carvajal Soto. "Unsupervised Anomaly Detection in Production Lines." In Machine Learning for Cyber Physical Systems, 18–25. Berlin, Heidelberg: Springer Berlin Heidelberg, 2018. http://dx.doi.org/10.1007/978-3-662-58485-9_3.
Full text
8
Eskin, Eleazar, Andrew Arnold, Michael Prerau, Leonid Portnoy, and Sal Stolfo. "A Geometric Framework for Unsupervised Anomaly Detection." In Advances in Information Security, 77–101. Boston, MA: Springer US, 2002. http://dx.doi.org/10.1007/978-1-4615-0953-0_4.
Full text
9
Syarif, Iwan, Adam Prugel-Bennett, and Gary Wills. "Unsupervised Clustering Approach for Network Anomaly Detection." In Networked Digital Technologies, 135–45. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012. http://dx.doi.org/10.1007/978-3-642-30507-8_13.
Full text
10
Weng, Lingxuan, Maohan Liang, Ruobin Gao, and Zhong Shuo Chen. "Deep Learning-Empowered Unsupervised Maritime Anomaly Detection." In Communications in Computer and Information Science, 189–202. Singapore: Springer Nature Singapore, 2023. http://dx.doi.org/10.1007/978-981-99-8178-6_15.
Full textConference papers on the topic "Unsupervised anomaly detection"
1
Blum, Ashlae. "A Temporal Approach to Unsupervised Anomaly Detection." In A Temporal Approach to Unsupervised Anomaly Detection. US DOE, 2021. http://dx.doi.org/10.2172/1825324.
Full text
2
Alnutefy, Suliman, and Ali Alsuwayh. "Unsupervised Anomaly Detection." In 4th International Conference on AI, Machine Learning and Applications. Academy & Industry Research Collaboration Center, 2024. http://dx.doi.org/10.5121/csit.2024.140210.
Full textAbstract:
This research focuses on Unsupervised Anomaly Detection using the "ambient_temperature_system_failure.csv" dataset from Numenta Anomaly Benchmark (NAB). The dataset contains time-series temperature readings from an industrial machine's sensor. The aim is to detect anomalies indicating system failures or aberrant behavior without labeled data. Various algorithms, such as K-means, Gaussian/Elliptic Envelopes, Markov Chain, Isolation Forest, One-Class SVM, and RNNs, are applied to analyze the temperature data. These algorithms are chosen for their ability to identify significant deviations in unlabeled datasets. The study explores how these techniques enhance anomaly understanding in time series data, relevant in manufacturing, healthcare, and finance. This research's novelty lies in employing unsupervised learning techniques on a real-world dataset and understanding theiradaptability in anomaly detection. The results are expected to contribute valuable insights to the field, showcasing the practicality and effectiveness of these algorithms across various scenarios.
3
Li, Tangqing, Zheng Wang, Siying Liu, and Wen-Yan Lin. "Deep Unsupervised Anomaly Detection." In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2021. http://dx.doi.org/10.1109/wacv48630.2021.00368.
Full text
4
Zhang, Zheng, and Liang Zhao. "Unsupervised Deep Subgraph Anomaly Detection (Extended Abstract)." In Thirty-Second International Joint Conference on Artificial Intelligence {IJCAI-23}. California: International Joint Conferences on Artificial Intelligence Organization, 2023. http://dx.doi.org/10.24963/ijcai.2023/730.
Full textAbstract:
Effectively mining anomalous subgraphs in networks is crucial for various applications, including disease outbreak detection, financial fraud detection, and activity monitoring in social networks. However, identifying anomalous subgraphs poses significant challenges due to their complex topological structures, high-dimensional attributes, multiple notions of anomalies, and the vast subgraph space within a given graph. Classical shallow models rely on handcrafted anomaly measure functions, limiting their applicability when prior knowledge is unavailable. Deep learning-based methods have shown promise in detecting node-level, edge-level, and graph-level anomalies, but subgraph-level anomaly detection remains under-explored due to difficulties in subgraph representation learning, supervision, and end-to-end anomaly quantification. To address these challenges, this paper introduces a novel deep framework named Anomalous Subgraph Autoencoder (AS-GAE). AS-GAE leverages an unsupervised and weakly supervised approach to extract anomalous subgraphs. It incorporates a location-aware graph autoencoder to uncover anomalous areas based on reconstruction mismatches and introduces a supermodular graph scoring function module to assign meaningful anomaly scores to subgraphs within the identified anomalous areas. Extensive experiments on synthetic and real-world datasets demonstrate the effectiveness of our proposed method.
5
Bekiroglu, Korkut, Ali Tekeoglu, Bruno Andriamanalimanana, Saumendra Sengupta, Chen-Fu Chiang, and Jorge Novillo. "Hankel-based Unsupervised Anomaly Detection." In 2020 American Control Conference (ACC). IEEE, 2020. http://dx.doi.org/10.23919/acc45564.2020.9147583.
Full text
6
Bang, Jaeho, Sungchul Kim, Ryan Rossi, Tong Yu, and Handong Zhao. "Interpretable Unsupervised Log Anomaly Detection." In 2023 IEEE International Conference on Big Data (BigData). IEEE, 2023. http://dx.doi.org/10.1109/bigdata59044.2023.10386852.
Full text
7
Zhang, Zheng, and Liang Zhao. "Unsupervised Deep Subgraph Anomaly Detection." In 2022 IEEE International Conference on Data Mining (ICDM). IEEE, 2022. http://dx.doi.org/10.1109/icdm54844.2022.00086.
Full text
8
Ye, Hangting, Zhining Liu, Xinyi Shen, Wei Cao, Shun Zheng, Xiaofan Gui, Huishuai Zhang, Yi Chang, and Jiang Bian. "UADB: Unsupervised Anomaly Detection Booster." In 2023 IEEE 39th International Conference on Data Engineering (ICDE). IEEE, 2023. http://dx.doi.org/10.1109/icde55515.2023.00199.
Full text
9
Chen, Xinqiang, Lumei Su, Guansen Deng, Mingyong Huang, Jiajun Wu, and Yanqing Peng. "Weak anomaly-reinforced autoencoder for unsupervised anomaly detection." In Thirteenth International Conference on Machine Vision, edited by Wolfgang Osten, Jianhong Zhou, and Dmitry P. Nikolaev. SPIE, 2021. http://dx.doi.org/10.1117/12.2587017.
Full text
10
Fine, Benjamin T. "Unsupervised anomaly detection with minimal sensing." In the 47th Annual Southeast Regional Conference. New York, New York, USA: ACM Press, 2009. http://dx.doi.org/10.1145/1566445.1566525.
Full text