Journal articles on the topic 'Machine learning, big data, anomaly detection, network monitoring'
To see the other types of publications on this topic, follow the link: Machine learning, big data, anomaly detection, network monitoring.
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the top 50 journal articles for your research on the topic 'Machine learning, big data, anomaly detection, network monitoring.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Browse journal articles on a wide variety of disciplines and organise your bibliography correctly.
1
Oprea, Simona-Vasilica, Adela Bâra, Florina Camelia Puican, and Ioan Cosmin Radu. "Anomaly Detection with Machine Learning Algorithms and Big Data in Electricity Consumption." Sustainability 13, no. 19 (October 2, 2021): 10963. http://dx.doi.org/10.3390/su131910963.
Full textAbstract:
When analyzing smart metering data, both reading errors and frauds can be identified. The purpose of this analysis is to alert the utility companies to suspicious consumption behavior that could be further investigated with on-site inspections or other methods. The use of Machine Learning (ML) algorithms to analyze consumption readings can lead to the identification of malfunctions, cyberattacks interrupting measurements, or physical tampering with smart meters. Fraud detection is one of the classical anomaly detection examples, as it is not easy to label consumption or transactional data. Furthermore, frauds differ in nature, and learning is not always possible. In this paper, we analyze large datasets of readings provided by smart meters installed in a trial study in Ireland by applying a hybrid approach. More precisely, we propose an unsupervised ML technique to detect anomalous values in the time series, establish a threshold for the percentage of anomalous readings from the total readings, and then label that time series as suspicious or not. Initially, we propose two types of algorithms for anomaly detection for unlabeled data: Spectral Residual-Convolutional Neural Network (SR-CNN) and an anomaly trained model based on martingales for determining variations in time-series data streams. Then, the Two-Class Boosted Decision Tree and Fisher Linear Discriminant analysis are applied on the previously processed dataset. By training the model, we obtain the required capabilities of detecting suspicious consumers proved by an accuracy of 90%, precision score of 0.875, and F1 score of 0.894.
2
Alnafessah, Ahmad, and Giuliano Casale. "Artificial neural networks based techniques for anomaly detection in Apache Spark." Cluster Computing 23, no. 2 (October 23, 2019): 1345–60. http://dx.doi.org/10.1007/s10586-019-02998-y.
Full textAbstract:
Abstract Late detection and manual resolutions of performance anomalies in Cloud Computing and Big Data systems may lead to performance violations and financial penalties. Motivated by this issue, we propose an artificial neural network based methodology for anomaly detection tailored to the Apache Spark in-memory processing platform. Apache Spark is widely adopted by industry because of its speed and generality, however there is still a shortage of comprehensive performance anomaly detection methods applicable to this platform. We propose an artificial neural networks driven methodology to quickly sift through Spark logs data and operating system monitoring metrics to accurately detect and classify anomalous behaviors based on the Spark resilient distributed dataset characteristics. The proposed method is evaluated against three popular machine learning algorithms, decision trees, nearest neighbor, and support vector machine, as well as against four variants that consider different monitoring datasets. The results prove that our proposed method outperforms other methods, typically achieving 98–99% F-scores, and offering much greater accuracy than alternative techniques to detect both the period in which anomalies occurred and their type.
3
Borghesi, Andrea, Andrea Bartolini, Michele Lombardi, Michela Milano, and Luca Benini. "Anomaly Detection Using Autoencoders in High Performance Computing Systems." Proceedings of the AAAI Conference on Artificial Intelligence 33 (July 17, 2019): 9428–33. http://dx.doi.org/10.1609/aaai.v33i01.33019428.
Full textAbstract:
Anomaly detection in supercomputers is a very difficult problem due to the big scale of the systems and the high number of components. The current state of the art for automated anomaly detection employs Machine Learning methods or statistical regression models in a supervised fashion, meaning that the detection tool is trained to distinguish among a fixed set of behaviour classes (healthy and unhealthy states).We propose a novel approach for anomaly detection in HighPerformance Computing systems based on a Machine (Deep) Learning technique, namely a type of neural network called autoencoder. The key idea is to train a set of autoencoders to learn the normal (healthy) behaviour of the supercomputer nodes and, after training, use them to identify abnormal conditions. This is different from previous approaches which where based on learning the abnormal condition, for which there are much smaller datasets (since it is very hard to identify them to begin with).We test our approach on a real supercomputer equipped with a fine-grained, scalable monitoring infrastructure that can provide large amount of data to characterize the system behaviour. The results are extremely promising: after the training phase to learn the normal system behaviour, our method is capable of detecting anomalies that have never been seen before with a very good accuracy (values ranging between 88% and 96%).
4
Albattah, Albatul, and Murad A. Rassam. "A Correlation-Based Anomaly Detection Model for Wireless Body Area Networks Using Convolutional Long Short-Term Memory Neural Network." Sensors 22, no. 5 (March 2, 2022): 1951. http://dx.doi.org/10.3390/s22051951.
Full textAbstract:
As the Internet of Healthcare Things (IoHT) concept emerges today, Wireless Body Area Networks (WBAN) constitute one of the most prominent technologies for improving healthcare services. WBANs are made up of tiny devices that can effectively enhance patient quality of life by collecting and monitoring physiological data and sending it to healthcare givers to assess the criticality of a patient and act accordingly. The collected data must be reliable and correct, and represent the real context to facilitate right and prompt decisions by healthcare personnel. Anomaly detection becomes a field of interest to ensure the reliability of collected data by detecting malicious data patterns that result due to various reasons such as sensor faults, error readings and possible malicious activities. Various anomaly detection solutions have been proposed for WBAN. However, existing detection approaches, which are mostly based on statistical and machine learning techniques, become ineffective in dealing with big data streams and novel context anomalous patterns in WBAN. Therefore, this paper proposed a model that employs the correlations that exist in the different physiological data attributes with the ability of the hybrid Convolutional Long Short-Term Memory (ConvLSTM) techniques to detect both simple point anomalies as well as contextual anomalies in the big data stream of WBAN. Experimental evaluations revealed that an average of 98% of F1-measure and 99% accuracy were reported by the proposed model on different subjects of the datasets compared to 64% achieved by both CNN and LSTM separately.
5
Chen, Naiyue, Yi Jin, Yinglong Li, and Luxin Cai. "Trust-based federated learning for network anomaly detection." Web Intelligence 19, no. 4 (January 20, 2022): 317–27. http://dx.doi.org/10.3233/web-210475.
Full textAbstract:
With the rapid development of social networks and the massive popularity of intelligent mobile terminals, network anomaly detection is becoming increasingly important. In daily work and life, edge nodes store a large number of network local connection data and audit data, which can be used to analyze network abnormal behavior. With the increasingly close network communication, the amount of network connection and other related data collected by each network terminal is increasing. Machine learning has become a classification method to analyze the features of big data in the network. Face to the problems of excessive data and long response time for network anomaly detection, we propose a trust-based Federated learning anomaly detection algorithm. We use the edge nodes to train the local data model, and upload the machine learning parameters to the central node. Meanwhile, according to the performance of edge nodes training, we set different weights to match the processing capacity of each terminal which will obtain faster convergence speed and better attack classification accuracy. The user’s private information will only be processed locally and will not be uploaded to the central server, which can reduce the risk of information disclosure. Finally, we compare the basic federated learning model and TFCNN algorithm on KDD Cup 99 dataset and MNIST dataset. The experimental results show that the TFCNN algorithm can improve accuracy and communication efficiency.
6
Do, ChoXuan, Nguyen Quang Dam, and Nguyen Tung Lam. "Optimization of network traffic anomaly detection using machine learning." International Journal of Electrical and Computer Engineering (IJECE) 11, no. 3 (June 1, 2021): 2360. http://dx.doi.org/10.11591/ijece.v11i3.pp2360-2370.
Full textAbstract:
In this paper, to optimize the process of detecting cyber-attacks, we choose to propose 2 main optimization solutions: Optimizing the detection method and optimizing features. Both of these two optimization solutions are to ensure the aim is to increase accuracy and reduce the time for analysis and detection. Accordingly, for the detection method, we recommend using the Random Forest supervised classification algorithm. The experimental results in section 4.1 have proven that our proposal that use the Random Forest algorithm for abnormal behavior detection is completely correct because the results of this algorithm are much better than some other detection algorithms on all measures. For the feature optimization solution, we propose to use some data dimensional reduction techniques such as information gain, principal component analysis, and correlation coefficient method. The results of the research proposed in our paper have proven that to optimize the cyber-attack detection process, it is not necessary to use advanced algorithms with complex and cumbersome computational requirements, it must depend on the monitoring data for selecting the reasonable feature extraction and optimization algorithm as well as the appropriate attack classification and detection algorithms.
7
Vajda, Daniel, Adrian Pekar, and Karoly Farkas. "Towards Machine Learning-based Anomaly Detection on Time-Series Data." Infocommunications journal 13, no. 1 (2021): 35–44. http://dx.doi.org/10.36244/icj.2021.1.5.
Full textAbstract:
The complexity of network infrastructures is exponentially growing. Real-time monitoring of these infrastructures is essential to secure their reliable operation. The concept of telemetry has been introduced in recent years to foster this process by streaming time-series data that contain feature-rich information concerning the state of network components. In this paper, we focus on a particular application of telemetry — anomaly detection on time-series data. We rigorously examined state-of-the-art anomaly detection methods. Upon close inspection of the methods, we observed that none of them suits our requirements as they typically face several limitations when applied on time-series data. This paper presents Alter-Re2, an improved version of ReRe, a state-of-the-art Long Short- Term Memory-based machine learning algorithm. Throughout a systematic examination, we demonstrate that by introducing the concepts of ageing and sliding window, the major limitations of ReRe can be overcome. We assessed the efficacy of Alter-Re2 using ten different datasets and achieved promising results. Alter-Re2 performs three times better on average when compared to ReRe.
8
Novoa-Paradela, David, Óscar Fontenla-Romero, and Bertha Guijarro-Berdiñas. "Adaptive Real-Time Method for Anomaly Detection Using Machine Learning." Proceedings 54, no. 1 (August 22, 2020): 38. http://dx.doi.org/10.3390/proceedings2020054038.
Full textAbstract:
Anomaly detection is a sub-area of machine learning that deals with the development of methods to distinguish among normal and anomalous data. Due to the frequent use of anomaly-detection systems in monitoring and the lack of methods capable of learning in real time, this research presents a new method that provides such online adaptability. The method bases its operation on the properties of scaled convex hulls. It begins building a convex hull, using a minimum set of data, that is adapted and subdivided along time to accurately fit the boundary of the normal class data. The model has online learning ability and its execution can be carried out in a distributed and parallel way, all of them interesting advantages when dealing with big datasets. The method has been compared to other state-of-the-art algorithms demonstrating its effectiveness.
9
Chimphlee, Siriporn, and Witcha Chimphlee. "Machine learning to improve the performance of anomaly-based network intrusion detection in big data." Indonesian Journal of Electrical Engineering and Computer Science 30, no. 2 (May 1, 2023): 1106. http://dx.doi.org/10.11591/ijeecs.v30.i2.pp1106-1119.
Full textAbstract:
With the rapid growth of digital technology communications are overwhelmed by network data traffic. The demand for the internet is growing every day in today's cyber world, raising concerns about network security. Big Data are a term that describes a vast volume of complicated data that is critical for evaluating network patterns and determining what has occurred in the network. Therefore, detecting attacks in a large network is challenging. Intrusion detection system (IDS) is a promising cybersecurity research field. In this paper, we proposed an efficient classification scheme for IDS, which is divided into two procedures, on the CSE-CIC-IDS-2018 dataset, data pre-processing techniques including under-sampling, feature selection, and classifier algorithms were used to assess and decide the best performing model to classify invaders. We have implemented and compared seven classifier machine learning algorithms with various criteria. This work explored the application of the random forest (RF) for feature selection in conjunction with machine learning (ML) techniques including linear regression (LR), k-Nearest Neighbor (k-NN), classification and regression trees (CART), Bayes, RF, multi layer perceptron (MLP), and XGBoost in order to implement IDSS. The experimental results show that the MLP algorithm in the most successful with best performance with evaluation matrix.
10
Káš, M., and F. F. Wamba. "Anomaly detection-based condition monitoring." Insight - Non-Destructive Testing and Condition Monitoring 64, no. 8 (August 1, 2022): 453–58. http://dx.doi.org/10.1784/insi.2022.64.8.453.
Full textAbstract:
The impact of an anomaly is domain-dependent. In a dataset of network activities, an anomaly can imply an intrusion attack. Other objectives of anomaly detection are industrial damage detection, data leak prevention, identifying security vulnerabilities or military surveillance. Anomalies are observations or a sequence of observations in which the distribution deviates remarkably from the general distribution of the whole dataset. A large majority of the dataset consists of normal (healthy) data points. The anomalies form only a very small part of the dataset. Anomaly detection is the technique used to find these observations and its methods are specific to the type of data. While there is a wide spectrum of anomaly detection approaches available today, it becomes increasingly difficult to keep track of all the techniques. In fact, it is not clear which of the three categories of detection methods, ie statistical approaches, machine learning approaches or deep learning approaches, is more appropriate to detect anomalies in time-series data, which are mainly used in industry. A typical industrial device has multi-dimensional characteristics. It is possible to measure voltage, current, active power, vibrations, rotational speed, temperature, pressure difference, etc, on such a device. Early detection of the anomalous behaviour of industrial devices can help reduce or prevent serious damage, which could lead to significant financial loss. This paper presents a summary of the methods used to detect anomalies in condition monitoring applications.
11
Preuveneers, Davy, Vera Rimmer, Ilias Tsingenopoulos, Jan Spooren, Wouter Joosen, and Elisabeth Ilie-Zudor. "Chained Anomaly Detection Models for Federated Learning: An Intrusion Detection Case Study." Applied Sciences 8, no. 12 (December 18, 2018): 2663. http://dx.doi.org/10.3390/app8122663.
Full textAbstract:
The adoption of machine learning and deep learning is on the rise in the cybersecurity domain where these AI methods help strengthen traditional system monitoring and threat detection solutions. However, adversaries too are becoming more effective in concealing malicious behavior amongst large amounts of benign behavior data. To address the increasing time-to-detection of these stealthy attacks, interconnected and federated learning systems can improve the detection of malicious behavior by joining forces and pooling together monitoring data. The major challenge that we address in this work is that in a federated learning setup, an adversary has many more opportunities to poison one of the local machine learning models with malicious training samples, thereby influencing the outcome of the federated learning and evading detection. We present a solution where contributing parties in federated learning can be held accountable and have their model updates audited. We describe a permissioned blockchain-based federated learning method where incremental updates to an anomaly detection machine learning model are chained together on the distributed ledger. By integrating federated learning with blockchain technology, our solution supports the auditing of machine learning models without the necessity to centralize the training data. Experiments with a realistic intrusion detection use case and an autoencoder for anomaly detection illustrate that the increased complexity caused by blockchain technology has a limited performance impact on the federated learning, varying between 5 and 15%, while providing full transparency over the distributed training process of the neural network. Furthermore, our blockchain-based federated learning solution can be generalized and applied to more sophisticated neural network architectures and other use cases.
12
Ahn, Hyojung, Han-Lim Choi, Minguk Kang, and SungTae Moon. "Learning-Based Anomaly Detection and Monitoring for Swarm Drone Flights." Applied Sciences 9, no. 24 (December 13, 2019): 5477. http://dx.doi.org/10.3390/app9245477.
Full textAbstract:
This paper addresses anomaly detection and monitoring for swarm drone flights. While the current practice of swarm flight typically relies on the operator’s naked eyes to monitor health of the multiple vehicles, this work proposes a machine learning-based framework to enable detection of abnormal behavior of a large number of flying drones on the fly. The method works in two steps: a sequence of two unsupervised learning procedures reduces the dimensionality of the real flight test data and labels them as normal and abnormal cases; then, a deep neural network classifier with one-dimensional convolution layers followed by fully connected multi-layer perceptron extracts the associated features and distinguishes the anomaly from normal conditions. The proposed anomaly detection scheme is validated on the real flight test data, highlighting its capability of online implementation.
13
Alkahtani, Hasan, Theyazn H. H. Aldhyani, and Mohammed Al-Yaari. "Adaptive Anomaly Detection Framework Model Objects in Cyberspace." Applied Bionics and Biomechanics 2020 (December 9, 2020): 1–14. http://dx.doi.org/10.1155/2020/6660489.
Full textAbstract:
Telecommunication has registered strong and rapid growth in the past decade. Accordingly, the monitoring of computers and networks is too complicated for network administrators. Hence, network security represents one of the biggest serious challenges that can be faced by network security communities. Taking into consideration the fact that e-banking, e-commerce, and business data will be shared on the computer network, these data may face a threat from intrusion. The purpose of this research is to propose a methodology that will lead to a high level and sustainable protection against cyberattacks. In particular, an adaptive anomaly detection framework model was developed using deep and machine learning algorithms to manage automatically-configured application-level firewalls. The standard network datasets were used to evaluate the proposed model which is designed for improving the cybersecurity system. The deep learning based on Long-Short Term Memory Recurrent Neural Network (LSTM-RNN) and machine learning algorithms namely Support Vector Machine (SVM), K-Nearest Neighbor (K-NN) algorithms were implemented to classify the Denial-of-Service attack (DoS) and Distributed Denial-of-Service (DDoS) attacks. The information gain method was applied to select the relevant features from the network dataset. These network features were significant to improve the classification algorithm. The system was used to classify DoS and DDoS attacks in four stand datasets namely KDD cup 199, NSL-KDD, ISCX, and ICI-ID2017. The empirical results indicate that the deep learning based on the LSTM-RNN algorithm has obtained the highest accuracy. The proposed system based on the LSTM-RNN algorithm produced the highest testing accuracy rate of 99.51% and 99.91% with respect to KDD Cup’99, NSL-KDD, ISCX, and ICI-Id2017 datasets, respectively. A comparative result analysis between the machine learning algorithms, namely SVM and KNN, and the deep learning algorithms based on the LSTM-RNN model is presented. Finally, it is concluded that the LSTM-RNN model is efficient and effective to improve the cybersecurity system for detecting anomaly-based cybersecurity.
14
Tang, Xiaoyu, Sijia Xu, and Hui Ye. "Labeling Expert: A New Multi-Network Anomaly Detection Architecture Based on LNN-RLSTM." Applied Sciences 13, no. 1 (December 31, 2022): 581. http://dx.doi.org/10.3390/app13010581.
Full textAbstract:
In network edge computing scenarios, close monitoring of network data and anomaly detection is critical for Internet services. Although a variety of anomaly detectors have been proposed by many scholars, few of these take into account the anomalies of the data in business logic. Expert labeling of business logic exceptions is also very important for detection. Most exception detection algorithms focus on problems, such as numerical exceptions, missed exceptions and false exceptions, but they ignore the existence of business logic exceptions, which brings a whole new challenge to exception detection. Moreover, anomaly detection in the context of big data is limited to the need to manually adjust detector parameters and thresholds, which is constrained by the physiological limits of operators. In this paper, a neural network algorithm based on the combination of Labeling Neural Network and Relevant Long Short-Term Memory Neural Network is proposed. This is a semi-supervised exception detection algorithm that can be readily extended with business logic exception types. The self-learning performance of this multi-network is better adapted to the big data anomaly detection scenario, which further improves the efficiency and accuracy of network data anomaly detection and considers business scenario-based anomaly data detection. The results show that the algorithm achieves 96% detection accuracy and 97% recall rate, which are consistent with the business logic anomaly fragments marked by experts. Both theoretical analysis and simulation experiments verify its effectiveness.
15
Thoidis, Iordanis, Marios Giouvanakis, and George Papanikolaou. "Semi-Supervised Machine Condition Monitoring by Learning Deep Discriminative Audio Features." Electronics 10, no. 20 (October 11, 2021): 2471. http://dx.doi.org/10.3390/electronics10202471.
Full textAbstract:
In this study, we aim to learn highly descriptive representations for a wide set of machinery sounds and exploit this knowledge to perform condition monitoring of mechanical equipment. We propose a comprehensive feature learning approach that operates on raw audio, by supervising the formation of salient audio embeddings in latent states of a deep temporal convolutional neural network. By fusing the supervised feature learning approach with an unsupervised deep one-class neural network, we are able to model the characteristics of each source and implicitly detect anomalies in different operational states of industrial machines. Moreover, we enable the exploitation of spatial audio information in the learning process, by formulating a novel front-end processing strategy for circular microphone arrays. Experimental results on the MIMII dataset demonstrate the effectiveness of the proposed method, reaching a state-of-the-art mean AUC score of 91.0%. Anomaly detection performance is significantly improved by incorporating multi-channel audio data in the feature extraction process, as well as training the convolutional neural network on the spatially invariant front-end. Finally, the proposed semi-supervised approach allows the concise modeling of normal machine conditions and accurately detects system anomalies, compared to existing anomaly detection methods.
16
Ramesh, Jayroop, Sakib Shahriar, A. R. Al-Ali, Ahmed Osman, and Mostafa F. Shaaban. "Machine Learning Approach for Smart Distribution Transformers Load Monitoring and Management System." Energies 15, no. 21 (October 27, 2022): 7981. http://dx.doi.org/10.3390/en15217981.
Full textAbstract:
Distribution transformers are an integral part of the power distribution system network and emerging smart grids. With the increasing dynamic service requirements of consumers, there is a higher likelihood of transformer failures due to overloading, feeder line faults, and ineffective cooling. As a consequence, their general longevity has been diminished, and the maintenance efforts of utility providers prove inadequate in efficiently monitoring and detecting transformer conditions. Existing Supervisory Control and Data Acquisition (SCADA) metering points are sparsely allocated in the network, making fault detection in feeder lines limited. To address these issues, this work proposes an IoT system for real-time distribution transformer load monitoring and anomaly detection. The monitoring system consists of a low-cost IoT gateway and sensor module which collects a three-phase load current profile, and oil levels/temperature from a distributed transformer network, specifically at the feeder side. The data are communicated through the publish/subscribe paradigm to a cloud IoT pipeline and stored in a cloud database after processing. An anomaly detection algorithm in the form of Isolation Forest is implemented to intelligently detect likely faults within a time window of 24 h prior. A mobile application was implemented to interact with the cloud database, visualize the real-time conditions of the transformers, and track them geographically. The proposed work can therefore reduce transformer maintenance costs with real-time monitoring and facilitate predictive fault analysis.
17
Poorvadevi, Dr R., Bodala Yaswanth Nikhil, and Darisi Venkata Sravan Kumar. "An Intelligent Data-Driven Model to Secure Intra- Vehicle Communications based on Machine Learning." International Journal for Research in Applied Science and Engineering Technology 10, no. 3 (March 31, 2022): 1329–34. http://dx.doi.org/10.22214/ijraset.2022.40863.
Full textAbstract:
Abstract: The depend on electric vehicles on either in vehicle or between-vehicle communications can cause big issues in the system. The model is constructed based on an better support vector machine model for difference finding based on the controller area network (CAN) bus protocol. In order to improve the capabilities of the model for fast mischievous attack detection and avoidance, a new optimization algorithm based on social spider (SSO) algorithm is developed which will emphasize the training process at. The model results on the real data sets tell the high performance, consistency hacking in the electric vehicles. Keywords: Electric Vehicle, Intra-Vehicle, Controller-AreaNetworks (CAN Bus), Anomaly Detection, Optimiz.
18
Laskar, Md Tahmid Rahman, Jimmy Xiangji Huang, Vladan Smetana, Chris Stewart, Kees Pouw, Aijun An, Stephen Chan, and Lei Liu. "Extending Isolation Forest for Anomaly Detection in Big Data via K-Means." ACM Transactions on Cyber-Physical Systems 5, no. 4 (October 31, 2021): 1–26. http://dx.doi.org/10.1145/3460976.
Full textAbstract:
Industrial Information Technology infrastructures are often vulnerable to cyberattacks. To ensure security to the computer systems in an industrial environment, it is required to build effective intrusion detection systems to monitor the cyber-physical systems (e.g., computer networks) in the industry for malicious activities. This article aims to build such intrusion detection systems to protect the computer networks from cyberattacks. More specifically, we propose a novel unsupervised machine learning approach that combines the K-Means algorithm with the Isolation Forest for anomaly detection in industrial big data scenarios. Since our objective is to build the intrusion detection system for the big data scenario in the industrial domain, we utilize the Apache Spark framework to implement our proposed model that was trained in large network traffic data (about 123 million instances of network traffic) stored in Elasticsearch. Moreover, we evaluate our proposed model on the live streaming data and find that our proposed system can be used for real-time anomaly detection in the industrial setup. In addition, we address different challenges that we face while training our model on large datasets and explicitly describe how these issues were resolved. Based on our empirical evaluation in different use cases for anomaly detection in real-world network traffic data, we observe that our proposed system is effective to detect anomalies in big data scenarios. Finally, we evaluate our proposed model on several academic datasets to compare with other models and find that it provides comparable performance with other state-of-the-art approaches.
19
Huang, Yu Liu, Junge, and Jihao Wang. "Environmental Safety Monitoring System Based on Microservice Architecture and Machine Learning." South Florida Journal of Development 2, no. 2 (June 4, 2021): 2894–902. http://dx.doi.org/10.46932/sfjdv2n2-133.
Full textAbstract:
The monitoring systems of various industries have various types and different structures. There are problems of “data chimney” and “information islands”. Monitoring data is difficult to be effectively utilized and cannot provide reliable data information to support for environmental security. In this end, an environment monitoring system based on micro-service architecture is designed. The information management and automatic monitoring business systems are unified into a flexible, robust and efficient system platform to adapt to the big data analysis and the mining applications. Using Hadoop to build environment monitoring big data platform, distributed storage, selective extraction and efficient calculation of the massive environment monitoring data can be achieved. By integrating the detection and monitoring data of the ecological environment and in-depth mining it, a neural network model is established to automatically identify potential safety hazards and recommend corresponding treatment measures, so to assist in the comprehensive research and scientific decision-making of environmental safety and promote intelligent management of safety.
20
Basora, Luis, Paloma Bry, Xavier Olive, and Floris Freeman. "Aircraft Fleet Health Monitoring with Anomaly Detection Techniques." Aerospace 8, no. 4 (April 7, 2021): 103. http://dx.doi.org/10.3390/aerospace8040103.
Full textAbstract:
Predictive maintenance has received considerable attention in the aviation industry where costs, system availability and reliability are major concerns. In spite of recent advances, effective health monitoring and prognostics for the scheduling of condition-based maintenance operations is still very challenging. The increasing availability of maintenance and operational data along with recent progress made in machine learning has boosted the development of data-driven prognostics and health management (PHM) models. In this paper, we describe the data workflow in place at an airline for the maintenance of an aircraft system and highlight the difficulties related to a proper labelling of the health status of such systems, resulting in a poor suitability of supervised learning techniques. We focus on investigating the feasibility and the potential of semi-supervised anomaly detection methods for the health monitoring of a real aircraft system. Proposed methods are evaluated on large volumes of real sensor data from a cooling unit system on a modern wide body aircraft from a major European airline. For the sake of confidentiality, data has been anonymized and only few technical and operational details about the system had been made available. We trained several deep neural network autoencoder architectures on nominal data and used the anomaly scores to calculate a health indicator. Results suggest that high anomaly scores are correlated with identified failures in the maintenance logs. Also, some situations see an increase in the anomaly score for several flights prior to the system’s failure, which paves a natural way for early fault identification.
21
Sireesha, P., Kongara Narmada, Kadurkapu Chandana, Govindu Badri, and Kalakonda Shirisha. "Detection of Diabetes Using 5G Network." International Journal for Research in Applied Science and Engineering Technology 10, no. 11 (November 30, 2022): 1656–60. http://dx.doi.org/10.22214/ijraset.2022.47622.
Full textAbstract:
bstract: Recent advances in wireless networking and big data technologies, such as 5G networks, medical big data analytics, and the Internet of Things, along with recent developments in wearable computing and artificial intelligence, are enabling the development and implementation of innovative diabetes monitoring systems and applications. Due to the life-long and systematic harm suffered by diabetes patients, it is critical to design effective methods for diagnosing and treating diabetes. Based on our comprehensive investigation, this article classifies those methods into Diabetes 1.0 and Diabetes 2.0, which exhibit deficiencies in terms of networking and intelligence. Thus, our goal is to design a sustainable, cost-effective, and intelligent diabetes diagnosis solution with personalized treatment. In this article, we first propose the 5G-Smart Diabetes system, which combines state-of-the-art technologies such as wearable 2.0, machine learning, and big data to generate comprehensive sensing and analysis for patients suffering from diabetes. Then we present the data sharing mechanism and personalized data analysis model for 5G-Smart Diabetes. Finally, we build a 5GSmart diabetes testbed that includes smart clothing, a smartphone, and a big data cloud. These experimental results show that our system can effectively provide patients with personalized diagnoses and treatment suggestions.
22
Mokhtari, Sohrab, Alireza Abbaspour, Kang K. Yen, and Arman Sargolzaei. "A Machine Learning Approach for Anomaly Detection in Industrial Control Systems Based on Measurement Data." Electronics 10, no. 4 (February 8, 2021): 407. http://dx.doi.org/10.3390/electronics10040407.
Full textAbstract:
Attack detection problems in industrial control systems (ICSs) are commonly known as a network traffic monitoring scheme for detecting abnormal activities. However, a network-based intrusion detection system can be deceived by attackers that imitate the system’s normal activity. In this work, we proposed a novel solution to this problem based on measurement data in the supervisory control and data acquisition (SCADA) system. The proposed approach is called measurement intrusion detection system (MIDS), which enables the system to detect any abnormal activity in the system even if the attacker tries to conceal it in the system’s control layer. A supervised machine learning model is generated to classify normal and abnormal activities in an ICS to evaluate the MIDS performance. A hardware-in-the-loop (HIL) testbed is developed to simulate the power generation units and exploit the attack dataset. In the proposed approach, we applied several machine learning models on the dataset, which show remarkable performances in detecting the dataset’s anomalies, especially stealthy attacks. The results show that the random forest is performing better than other classifier algorithms in detecting anomalies based on measured data in the testbed.
23
Ye, Jiaxing, Yuichi Kurashima, Takeshi Kobayashi, Hiroshi Tsuda, Teruyoshi Takahara, and Wataru Sakurai. "An Efficient In-Situ Debris Flow Monitoring System over a Wireless Accelerometer Network." Remote Sensing 11, no. 13 (June 26, 2019): 1512. http://dx.doi.org/10.3390/rs11131512.
Full textAbstract:
Debris flow disasters pose a serious threat to public safety in many areas all over the world, and it may cause severe consequences, including losses, injuries, and fatalities. With the emergence of deep learning and increased computation powers, nowadays, machine learning methods are being broadly acknowledged as a feasible solution to tackle the massive data generated from geo-informatics and sensing platforms to distill adequate information in the context of disaster monitoring. Aiming at detection of debris flow occurrences in a mountainous area of Sakurajima, Japan, this study demonstrates an efficient in-situ monitoring system which employs state-of-the-art machine learning techniques to exploit continuous monitoring data collected by a wireless accelerometer sensor network. Concretely, a two-stage data analysis process had been adopted, which consists of anomaly detection and debris flow event identification. The system had been validated with real data and generated favorable detection precision. Compared to other debris flow monitoring system, the proposed solution renders a batch of substantive merits, such as low-cost, high accuracy, and fewer maintenance efforts. Moreover, the presented data investigation scheme can be readily extended to deal with multi-modal data for more accurate debris monitoring, and we expect to expend addition sensory measurements shortly.
24
El-Khchine, Radouane, Amine Amar, Zine Elabidine Guennoun, Charaf Bensouda, and Youness Laaroussi. "Machine Learning for Supply Chain’s Big Data: State of the art and application to Social Networks’ data." MATEC Web of Conferences 200 (2018): 00015. http://dx.doi.org/10.1051/matecconf/201820000015.
Full textAbstract:
In the context of today ’s pattern of globalization and a huge amount of information, a smart supply management chain is required. Naturally, statistics and operations research are used for optimizing supply and demand objectives. However, the new context brings out new opportunities at descriptive, predictive and prescriptive levels for supply chain network design, logistics and distribution and strategic sourcing. The key question is still how to capture and to use information. One striking example can be taken from social media, where their use allow to gain insight into the perception of consumers and to capture a real time overview of consumer reactions, regarding one or more specific events. In this regard, different modern approaches, such as IoT or Quantum neural network, are developed. In the same line of thought, we propose an analytic approach, based on KNN, Logistic Regression and SVM with the use of Twitter data in chicken supply chain management. Results identify the main concerns related to chicken products and allow to the development of a consumer-centric supply chain. The proposed approach can be extended to other topics such as anomaly detection and codification of customer intelligence.
25
Karaçay, Leyli, Erkay Savaş, and Halit Alptekin. "Intrusion Detection Over Encrypted Network Data." Computer Journal 63, no. 4 (November 17, 2019): 604–19. http://dx.doi.org/10.1093/comjnl/bxz111.
Full textAbstract:
Abstract Effective protection against cyber-attacks requires constant monitoring and analysis of system data in an IT infrastructure, such as log files and network packets, which may contain private and sensitive information. Security operation centers (SOC), which are established to detect, analyze and respond to cyber-security incidents, often utilize detection models either for known types of attacks or for anomaly and applies them to the system data for detection. SOC are also motivated to keep their models private to capitalize on the models that are their propriety expertise, and to protect their detection strategies against adversarial machine learning. In this paper, we develop a protocol for privately evaluating detection models on the system data, in which privacy of both the system data and detection models is protected and information leakage is either prevented altogether or quantifiably decreased. Our main approach is to provide an end-to-end encryption for the system data and detection models utilizing lattice-based cryptography that allows homomorphic operations over ciphertext. We employ recent data sets in our experiments which demonstrate that the proposed privacy-preserving intrusion detection system is feasible in terms of execution times and bandwidth requirements and reliable in terms of accuracy.
26
Diro, Abebe, Naveen Chilamkurti, Van-Doan Nguyen, and Will Heyne. "A Comprehensive Study of Anomaly Detection Schemes in IoT Networks Using Machine Learning Algorithms." Sensors 21, no. 24 (December 13, 2021): 8320. http://dx.doi.org/10.3390/s21248320.
Full textAbstract:
The Internet of Things (IoT) consists of a massive number of smart devices capable of data collection, storage, processing, and communication. The adoption of the IoT has brought about tremendous innovation opportunities in industries, homes, the environment, and businesses. However, the inherent vulnerabilities of the IoT have sparked concerns for wide adoption and applications. Unlike traditional information technology (I.T.) systems, the IoT environment is challenging to secure due to resource constraints, heterogeneity, and distributed nature of the smart devices. This makes it impossible to apply host-based prevention mechanisms such as anti-malware and anti-virus. These challenges and the nature of IoT applications call for a monitoring system such as anomaly detection both at device and network levels beyond the organisational boundary. This suggests an anomaly detection system is strongly positioned to secure IoT devices better than any other security mechanism. In this paper, we aim to provide an in-depth review of existing works in developing anomaly detection solutions using machine learning for protecting an IoT system. We also indicate that blockchain-based anomaly detection systems can collaboratively learn effective machine learning models to detect anomalies.
27
Ibrahim, Juma, and Slavko Gajin. "Entropy-based network traffic anomaly classification method resilient to deception." Computer Science and Information Systems, no. 00 (2021): 45. http://dx.doi.org/10.2298/csis201229045i.
Full textAbstract:
Entropy-based network traffic anomaly detection techniques are attractive due to their simplicity and applicability in a real-time network environment. Even though flow data provide only a basic set of information about network communications, they are suitable for efficient entropy-based anomaly detection techniques. However, a recent work reported a serious weakness of the general entropy-based anomaly detection related to its susceptibility to deception by adding spoofed data that camouflage the anomaly. Moreover, techniques for further classification of the anomalies mostly rely on machine learning, which involves additional complexity. We address these issues by providing two novel approaches. Firstly, we propose an efficient protection mechanism against entropy deception, which is based on the analysis of changes in different entropy types, namely Shannon, R?nyi, and Tsallis entropies, and monitoring the number of distinct elements in a feature distribution as a new detection metric. The proposed approach makes the entropy techniques more reliable. Secondly, we have extended the existing entropy-based anomaly detection approach with the anomaly classification method. Based on a multivariate analysis of the entropy changes of multiple features as well as aggregation by complex feature combinations, entropy-based anomaly classification rules were proposed and successfully verified through experiments. Experimental results are provided to validate the feasibility of the proposed approach for practical implementation of efficient anomaly detection and classification method in the general real-life network environment.
28
Latif, Zohaib, Qasim Umer, Choonhwa Lee, Kashif Sharif, Fan Li, and Sujit Biswas. "A Machine Learning-Based Anomaly Prediction Service for Software-Defined Networks." Sensors 22, no. 21 (November 2, 2022): 8434. http://dx.doi.org/10.3390/s22218434.
Full textAbstract:
Software-defined networking (SDN) has gained tremendous growth and can be exploited in different network scenarios, from data centers to wide-area 5G networks. It shifts control logic from the devices to a centralized entity (programmable controller) for efficient traffic monitoring and flow management. A software-based controller enforces rules and policies on the requests sent by forwarding elements; however, it cannot detect anomalous patterns in the network traffic. Due to this, the controller may install the flow rules against the anomalies, reducing the overall network performance. These anomalies may indicate threats to the network and decrease its performance and security. Machine learning (ML) approaches can identify such traffic flow patterns and predict the systems’ impending threats. We propose an ML-based service to predict traffic anomalies for software-defined networks in this work. We first create a large dataset for network traffic by modeling a programmable data center with a signature-based intrusion-detection system. The feature vectors are pre-processed and are constructed against each flow request by the forwarding element. Then, we input the feature vector of each request to a machine learning classifier for training to predict anomalies. Finally, we use the holdout cross-validation technique to evaluate the proposed approach. The evaluation results specify that the proposed approach is highly accurate. In contrast to baseline approaches (random prediction and zero rule), the performance improvement of the proposed approach in average accuracy, precision, recall, and f-measure is (54.14%, 65.30%, 81.63%, and 73.70%) and (4.61%, 11.13%, 9.45%, and 10.29%), respectively.
29
Christyawan, Tomi Yahya, Ahmad Afif Supianto, and Wayan Firdaus Mahmudy. "Anomaly-based intrusion detector system using restricted growing self organizing map." Indonesian Journal of Electrical Engineering and Computer Science 13, no. 3 (March 1, 2019): 919. http://dx.doi.org/10.11591/ijeecs.v13.i3.pp919-926.
Full textAbstract:
<p><span>The rapid development of internet and network technology followed by malicious threats and attacks on networks and computers. Intrusion detection system (IDS) was developed to solve that problems. The development of IDS using machine learning is needed for classifying the attacks. One method of the classification is Self-Organizing Map (SOM). SOM able to perform classification and visualization in learning process to gain new knowledge. However, the SOM has less efficient in learning process when applied in Big Data. This study proposes Restricted Growing SOM method with clustering reference vector (RGSOM-CRV) and Parallel RGSOM-CRV to improve SOM efficiency in classification with accuracy consideration to solve Big Data problem. Growing process in RGSOM is restricted by maximum nodes and growing threshold, the reupdate weight process will update unused reference vector when map size already maximum, these two processes solve the consuming time of regular GSOM. From the results of this research against KDD Cup 1999 dataset, proposed method Parallel RGSOM-CRV able to give 91.86% accuracy, 20.58% false alarm rate, 95.32% recall or detection rate, and precision is 94.35% and time consuming is outperform than regular Growing SOM. This proposed method is very promising to handle big data problems compared with other methods.</span></p>
30
Li, Zhi, Fei Fei, and Guanglie Zhang. "Edge-to-Cloud IIoT for Condition Monitoring in Manufacturing Systems with Ubiquitous Smart Sensors." Sensors 22, no. 15 (August 7, 2022): 5901. http://dx.doi.org/10.3390/s22155901.
Full textAbstract:
The Industrial Internet of Things (IIoT) connects industrial assets to ubiquitous smart sensors and actuators to enhance manufacturing and industrial processes. Data-driven condition monitoring is an essential technology for intelligent manufacturing systems to identify anomalies from malfunctioning equipment, prevent unplanned downtime, and reduce the operation costs by predictive maintenance without interrupting normal machine operations. However, data-driven condition monitoring requires massive data collected from smart sensors to be transmitted to the cloud for further processing, thereby contributing to network congestion and affecting the network performance. Furthermore, unbalanced training data with very few labelled anomalies limit supervised learning models because of the lack of sufficient fault data for the training process in anomaly detection algorithms. To address these issues, we proposed an IIoT-based condition monitoring system with an edge-to-cloud architecture and computed the relative wavelet energy as feature vectors on the edge layer to reduce the network traffic overhead. We also proposed an unsupervised deep long short-term memory (LSTM) network module for anomaly detection. We implemented the proposed IIoT condition monitoring system for a manufacturing machine in a real shop site to evaluate our proposed solution. Our experimental results verify the effectiveness of our approach which can not only reduce the network traffic overhead for the IIoT but also detect anomalies accurately.
31
Thaseen, Ikram Sumaiya, Vanitha Mohanraj, Sakthivel Ramachandran, Kishore Sanapala, and Sang-Soo Yeo. "A Hadoop Based Framework Integrating Machine Learning Classifiers for Anomaly Detection in the Internet of Things." Electronics 10, no. 16 (August 13, 2021): 1955. http://dx.doi.org/10.3390/electronics10161955.
Full textAbstract:
In recent years, different variants of the botnet are targeting government, private organizations and there is a crucial need to develop a robust framework for securing the IoT (Internet of Things) network. In this paper, a Hadoop based framework is proposed to identify the malicious IoT traffic using a modified Tomek-link under-sampling integrated with automated Hyper-parameter tuning of machine learning classifiers. The novelty of this paper is to utilize a big data platform for benchmark IoT datasets to minimize computational time. The IoT benchmark datasets are loaded in the Hadoop Distributed File System (HDFS) environment. Three machine learning approaches namely naive Bayes (NB), K-nearest neighbor (KNN), and support vector machine (SVM) are used for categorizing IoT traffic. Artificial immune network optimization is deployed during cross-validation to obtain the best classifier parameters. Experimental analysis is performed on the Hadoop platform. The average accuracy of 99% and 90% is obtained for BoT_IoT and ToN_IoT datasets. The accuracy difference in ToN-IoT dataset is due to the huge number of data samples captured at the edge layer and fog layer. However, in BoT-IoT dataset only 5% of the training and test samples from the complete dataset are considered for experimental analysis as released by the dataset developers. The overall accuracy is improved by 19% in comparison with state-of-the-art techniques. The computational times for the huge datasets are reduced by 3–4 hours through Map Reduce in HDFS.
32
Apostol, Elena-Simona, Ciprian-Octavian Truică, Florin Pop, and Christian Esposito. "Change Point Enhanced Anomaly Detection for IoT Time Series Data." Water 13, no. 12 (June 10, 2021): 1633. http://dx.doi.org/10.3390/w13121633.
Full textAbstract:
Due to the exponential growth of the Internet of Things networks and the massive amount of time series data collected from these networks, it is essential to apply efficient methods for Big Data analysis in order to extract meaningful information and statistics. Anomaly detection is an important part of time series analysis, improving the quality of further analysis, such as prediction and forecasting. Thus, detecting sudden change points with normal behavior and using them to discriminate between abnormal behavior, i.e., outliers, is a crucial step used to minimize the false positive rate and to build accurate machine learning models for prediction and forecasting. In this paper, we propose a rule-based decision system that enhances anomaly detection in multivariate time series using change point detection. Our architecture uses a pipeline that automatically manages to detect real anomalies and remove the false positives introduced by change points. We employ both traditional and deep learning unsupervised algorithms, in total, five anomaly detection and five change point detection algorithms. Additionally, we propose a new confidence metric based on the support for a time series point to be an anomaly and the support for the same point to be a change point. In our experiments, we use a large real-world dataset containing multivariate time series about water consumption collected from smart meters. As an evaluation metric, we use Mean Absolute Error (MAE). The low MAE values show that the algorithms accurately determine anomalies and change points. The experimental results strengthen our assumption that anomaly detection can be improved by determining and removing change points as well as validates the correctness of our proposed rules in real-world scenarios. Furthermore, the proposed rule-based decision support systems enable users to make informed decisions regarding the status of the water distribution network and perform effectively predictive and proactive maintenance.
33
Naveed, Muhammad, Fahim Arif, Syed Muhammad Usman, Aamir Anwar, Myriam Hadjouni, Hela Elmannai, Saddam Hussain, Syed Sajid Ullah, and Fazlullah Umar. "A Deep Learning-Based Framework for Feature Extraction and Classification of Intrusion Detection in Networks." Wireless Communications and Mobile Computing 2022 (August 8, 2022): 1–11. http://dx.doi.org/10.1155/2022/2215852.
Full textAbstract:
An intrusion detection system, often known as an IDS, is extremely important for preventing attacks on a network, violating network policies, and gaining unauthorized access to a network. The effectiveness of IDS is highly dependent on data preprocessing techniques and classification models used to enhance accuracy and reduce model training and testing time. For the purpose of anomaly identification, researchers have developed several machine learning and deep learning-based algorithms; nonetheless, accurate anomaly detection with low test and train times remains a challenge. Using a hybrid feature selection approach and a deep neural network- (DNN-) based classifier, the authors of this research suggest an enhanced intrusion detection system (IDS). In order to construct a subset of reduced and optimal features that may be used for classification, a hybrid feature selection model that consists of three methods, namely, chi square, ANOVA, and principal component analysis (PCA), is applied. These methods are referred to as “the big three.” On the NSL-KDD dataset, the suggested model receives training and is then evaluated. The proposed method was successful in achieving the following results: a reduction of input data by 40%, an average accuracy of 99.73%, a precision score of 99.75%, an F1 score of 99.72%, and an average training and testing time of 138% and 2.7 seconds, respectively. The findings of the experiments demonstrate that the proposed model is superior to the performance of the other comparison approaches.
34
Munir, Mohsin, Shoaib Ahmed Siddiqui, Muhammad Ali Chattha, Andreas Dengel, and Sheraz Ahmed. "FuseAD: Unsupervised Anomaly Detection in Streaming Sensors Data by Fusing Statistical and Deep Learning Models." Sensors 19, no. 11 (May 29, 2019): 2451. http://dx.doi.org/10.3390/s19112451.
Full textAbstract:
The need for robust unsupervised anomaly detection in streaming data is increasing rapidly in the current era of smart devices, where enormous data are gathered from numerous sensors. These sensors record the internal state of a machine, the external environment, and the interaction of machines with other machines and humans. It is of prime importance to leverage this information in order to minimize downtime of machines, or even avoid downtime completely by constant monitoring. Since each device generates a different type of streaming data, it is normally the case that a specific kind of anomaly detection technique performs better than the others depending on the data type. For some types of data and use-cases, statistical anomaly detection techniques work better, whereas for others, deep learning-based techniques are preferred. In this paper, we present a novel anomaly detection technique, FuseAD, which takes advantage of both statistical and deep-learning-based approaches by fusing them together in a residual fashion. The obtained results show an increase in area under the curve (AUC) as compared to state-of-the-art anomaly detection methods when FuseAD is tested on a publicly available dataset (Yahoo Webscope benchmark). The obtained results advocate that this fusion-based technique can obtain the best of both worlds by combining their strengths and complementing their weaknesses. We also perform an ablation study to quantify the contribution of the individual components in FuseAD, i.e., the statistical ARIMA model as well as the deep-learning-based convolutional neural network (CNN) model.
35
Manzano Sanchez, Ricardo Alejandro, Marzia Zaman, Nishith Goel, Kshirasagar Naik, and Rohit Joshi. "Towards Developing a Robust Intrusion Detection Model Using Hadoop–Spark and Data Augmentation for IoT Networks." Sensors 22, no. 20 (October 12, 2022): 7726. http://dx.doi.org/10.3390/s22207726.
Full textAbstract:
In recent years, anomaly detection and machine learning for intrusion detection systems have been used to detect anomalies on Internet of Things networks. These systems rely on machine and deep learning to improve the detection accuracy. However, the robustness of the model depends on the number of datasamples available, quality of the data, and the distribution of the data classes. In the present paper, we focused specifically on the amount of data and class imbalanced since both parameters are key in IoT due to the fact that network traffic is increasing exponentially. For this reason, we propose a framework that uses a big data methodology with Hadoop–Spark to train and test multi-class and binary classification with one-vs-rest strategy for intrusion detection using the entire BoT IoT dataset. Thus, we evaluate all the algorithms available in Hadoop–Spark in terms of accuracy and processing time. In addition, since the BoT IoT dataset used is highly imbalanced, we also improve the accuracy for detecting minority classes by generating more datasamples using a Conditional Tabular Generative Adversarial Network (CTGAN). In general, our proposed model outperforms other published models including our previous model. Using our proposed methodology, the F1-score of one of the minority class, i.e., Theft attack was improved from 42% to 99%.
36
Elia, Domenico, Gioacchino Vino, Giacinto Donvito, and Marica Antonacci. "Developing a monitoring system for Cloud-based distributed data-centers." EPJ Web of Conferences 214 (2019): 08012. http://dx.doi.org/10.1051/epjconf/201921408012.
Full textAbstract:
Nowadays more and more datacenters cooperate each others to achieve a common and more complex goal. New advanced functionalities are required to support experts during recovery and managing activities, like anomaly detection and fault pattern recognition. The proposed solution provides an active support to problem solving for datacenter management teams by providing automatically the root-cause of detected anomalies. The project has been developed in Bari using the datacenter ReCaS as testbed. Big Data solutions have been selected to properly handle the complexity and size of the data. Features like open source, big community, horizontal scalability and high availability have been considered and tools belonging to the Hadoop ecosystem have been selected. The collected information is sent to a combination of Apache Flume and Apache Kafka, used as transport layer, in turn delivering data to databases and processing components. Apache Spark has been selected as analysis component. Different kind of databases have been considered in order to satisfy multiple requirements: Hadoop Distributed File System, Neo4j, InfluxDB and Elasticsearch. Grafana and Kibana are used to show data in a dedicated dashboards. The Root-cause analysis engine has been implemented using custom machine learning algorithms. Finally, results are forwarded to experts by email or Slack, using Riemann.
37
Imran, Faisal Jamil, and Dohyeun Kim. "An Ensemble of a Prediction and Learning Mechanism for Improving Accuracy of Anomaly Detection in Network Intrusion Environments." Sustainability 13, no. 18 (September 8, 2021): 10057. http://dx.doi.org/10.3390/su131810057.
Full textAbstract:
The connectivity of our surrounding objects to the internet plays a tremendous role in our daily lives. Many network applications have been developed in every domain of life, including business, healthcare, smart homes, and smart cities, to name a few. As these network applications provide a wide range of services for large user groups, the network intruders are prone to developing intrusion skills for attack and malicious compliance. Therefore, safeguarding network applications and things connected to the internet has always been a point of interest for researchers. Many studies propose solutions for intrusion detection systems and intrusion prevention systems. Network communities have produced benchmark datasets available for researchers to improve the accuracy of intrusion detection systems. The scientific community has presented data mining and machine learning-based mechanisms to detect intrusion with high classification accuracy. This paper presents an intrusion detection system based on the ensemble of prediction and learning mechanisms to improve anomaly detection accuracy in a network intrusion environment. The learning mechanism is based on automated machine learning, and the prediction model is based on the Kalman filter. Performance analysis of the proposed intrusion detection system is evaluated using publicly available intrusion datasets UNSW-NB15 and CICIDS2017. The proposed model-based intrusion detection accuracy for the UNSW-NB15 dataset is 98.801 percent, and the CICIDS2017 dataset is 97.02 percent. The performance comparison results show that the proposed ensemble model-based intrusion detection significantly improves the intrusion detection accuracy.
38
Mitiche, Imene, Tony McGrail, Philip Boreham, Alan Nesbitt, and Gordon Morison. "Data-Driven Anomaly Detection in High-Voltage Transformer Bushings with LSTM Auto-Encoder." Sensors 21, no. 21 (November 8, 2021): 7426. http://dx.doi.org/10.3390/s21217426.
Full textAbstract:
The reliability and health of bushings in high-voltage (HV) power transformers is essential in the power supply industry, as any unexpected failure can cause power outage leading to heavy financial losses. The challenge is to identify the point at which insulation deterioration puts the bushing at an unacceptable risk of failure. By monitoring relevant measurements we can trace any change that occurs and may indicate an anomaly in the equipment’s condition. In this work we propose a machine-learning-based method for real-time anomaly detection in current magnitude and phase angle from three bushing taps. The proposed method is fast, self-supervised and flexible. It consists of a Long Short-Term Memory Auto-Encoder (LSTMAE) network which learns the normal current and phase measurements of the bushing and detects any point when these measurements change based on the Mean Absolute Error (MAE) metric evaluation. This approach was successfully evaluated using real-world data measured from HV transformer bushings where anomalous events have been identified.
39
Rashid, A. N. M. Bazlur, Mohiuddin Ahmed, and Al-Sakib Khan Pathan. "Infrequent Pattern Detection for Reliable Network Traffic Analysis Using Robust Evolutionary Computation." Sensors 21, no. 9 (April 25, 2021): 3005. http://dx.doi.org/10.3390/s21093005.
Full textAbstract:
While anomaly detection is very important in many domains, such as in cybersecurity, there are many rare anomalies or infrequent patterns in cybersecurity datasets. Detection of infrequent patterns is computationally expensive. Cybersecurity datasets consist of many features, mostly irrelevant, resulting in lower classification performance by machine learning algorithms. Hence, a feature selection (FS) approach, i.e., selecting relevant features only, is an essential preprocessing step in cybersecurity data analysis. Despite many FS approaches proposed in the literature, cooperative co-evolution (CC)-based FS approaches can be more suitable for cybersecurity data preprocessing considering the Big Data scenario. Accordingly, in this paper, we have applied our previously proposed CC-based FS with random feature grouping (CCFSRFG) to a benchmark cybersecurity dataset as the preprocessing step. The dataset with original features and the dataset with a reduced number of features were used for infrequent pattern detection. Experimental analysis was performed and evaluated using 10 unsupervised anomaly detection techniques. Therefore, the proposed infrequent pattern detection is termed Unsupervised Infrequent Pattern Detection (UIPD). Then, we compared the experimental results with and without FS in terms of true positive rate (TPR). Experimental analysis indicates that the highest rate of TPR improvement was by cluster-based local outlier factor (CBLOF) of the backdoor infrequent pattern detection, and it was 385.91% when using FS. Furthermore, the highest overall infrequent pattern detection TPR was improved by 61.47% for all infrequent patterns using clustering-based multivariate Gaussian outlier score (CMGOS) with FS.
40
Minea, Marius, Cătălin Marian Dumitrescu, and Viviana Laetitia Minea. "Intelligent Network Applications Monitoring and Diagnosis Employing Software Sensing and Machine Learning Solutions." Sensors 21, no. 15 (July 25, 2021): 5036. http://dx.doi.org/10.3390/s21155036.
Full textAbstract:
The article presents a research in the field of complex sensing, detection, and recovery of communications networks applications and hardware, in case of failures, maloperations, or unauthorized intrusions. A case study, based on Davis AI engine operation versus human maintenance operation is performed on the efficiency of artificial intelligence agents in detecting faulty operation, in the context of growing complexity of communications networks, and the perspective of future development of internet of things, big data, smart cities, and connected vehicles. (*). In the second part of the article, a new solution is proposed for the detection of applications faults or unauthorized intrusions in traffic of communications networks. The first objective of the proposed method is to propose an approach for predicting time series. This approach is based on a multi-resolution decomposition of the signals employing the undecimate wavelet transform (UWT). The second approach for assessing traffic flow is based on the analysis of long-range dependence (LRD) (for this case, a long-term dependence). Estimating the degree of long-range dependence is performed by estimating the Hurst parameter of the analyzed time series. This is a relatively new statistical concept in communications traffic analysis and can be implemented using UWT. This property has important implications for network performance, design, and sizing. The presence of long-range dependency in network traffic is assumed to have a significant impact on network performance, and the occurrence of LRD can be the result of faults that occur during certain periods. The strategy chosen for this purpose is based on long-term dependence on traffic, and for the prediction of faults occurrence, a predictive control model (MPC) is proposed, combined with a neural network with radial function (RBF). It is demonstrated via simulations that, in the case of communications traffic, time location is the most important feature of the proposed algorithm.
41
Eketnova, Yu M. "Comparative Analysis of Machine learning Methods to Identify signs of suspicious Transactions of Credit Institutions and Their Clients." Finance: Theory and Practice 25, no. 5 (October 28, 2021): 186–99. http://dx.doi.org/10.26794/2587-5671-2020-25-5-186-199.
Full textAbstract:
In the field of financial monitoring, it is necessary to promptly obtain objective assessments of economic entities (in particular, credit institutions) for effective decision-making. Automation of the process of identifying unscrupulous credit institutions based on machine learning methods will allow regulatory authorities to quickly identify and suppress illegal activities. The aim of the research is to substantiate the possibilities of using machine learning methods and algorithms for the automatic identification of unscrupulous credit institutions. It is required to select a mathematical toolkit for analyzing data on credit institutions, which allows tracking the involvement of a bank in money laundering processes. The paper provides a comparative analysis of the results of processing data on the activities of credit institutions using classification methods — logistic regression, decision trees. The author applies support vector machine and neural network methods, Bayesian networks (Two-Class Bayes Point Machine), and anomaly search — an algorithm of a One-Class Support Vector Machine and a PCA-Based Anomaly Detection algorithm. The study presents the results of solving the problem of classifying credit institutions in terms of possible involvement in money laundering processes, the results of analyzing data on the activities of credit institutions by methods of detecting anomalies. A comparative analysis of the results obtained using various modern algorithms for the classification and search for anomalies is carried out. The author concluded that the PCA-Based Anomaly Detection algorithm showed more accurate results compared to the One-Class Support Vector Machine algorithm. Of the considered classification algorithms, the most accurate results were shown by the Two-Class Boosted Decision Tree (AdaBoost) algorithm. The research results can be used by the Bank of Russia and Rosfinmonitoring to automate the identification of unscrupulous credit institutions
42
Alzahrani, Abdulsalam O., and Mohammed J. F. Alenazi. "Designing a Network Intrusion Detection System Based on Machine Learning for Software Defined Networks." Future Internet 13, no. 5 (April 28, 2021): 111. http://dx.doi.org/10.3390/fi13050111.
Full textAbstract:
Software-defined Networking (SDN) has recently developed and been put forward as a promising and encouraging solution for future internet architecture. Managed, the centralized and controlled network has become more flexible and visible using SDN. On the other hand, these advantages bring us a more vulnerable environment and dangerous threats, causing network breakdowns, systems paralysis, online banking frauds and robberies. These issues have a significantly destructive impact on organizations, companies or even economies. Accuracy, high performance and real-time systems are essential to achieve this goal successfully. Extending intelligent machine learning algorithms in a network intrusion detection system (NIDS) through a software-defined network (SDN) has attracted considerable attention in the last decade. Big data availability, the diversity of data analysis techniques, and the massive improvement in the machine learning algorithms enable the building of an effective, reliable and dependable system for detecting different types of attacks that frequently target networks. This study demonstrates the use of machine learning algorithms for traffic monitoring to detect malicious behavior in the network as part of NIDS in the SDN controller. Different classical and advanced tree-based machine learning techniques, Decision Tree, Random Forest and XGBoost are chosen to demonstrate attack detection. The NSL-KDD dataset is used for training and testing the proposed methods; it is considered a benchmarking dataset for several state-of-the-art approaches in NIDS. Several advanced preprocessing techniques are performed on the dataset in order to extract the best form of the data, which produces outstanding results compared to other systems. Using just five out of 41 features of NSL-KDD, a multi-class classification task is conducted by detecting whether there is an attack and classifying the type of attack (DDoS, PROBE, R2L, and U2R), accomplishing an accuracy of 95.95%.
43
Lu, Jiazhong, Weina Niu, Xiaolei Liu, Teng Hu, and Xiaosong Zhang. "A Lockable Abnormal Electromagnetic Signal Joint Detection Algorithm." International Journal of Pattern Recognition and Artificial Intelligence 33, no. 13 (December 15, 2019): 1958009. http://dx.doi.org/10.1142/s0218001419580096.
Full textAbstract:
With the development of computers and network technologies, network security has gradually become a global problem. Network security defenses need to be carried out not only on the Internet, but also on other communication media, such as electromagnetic signals. Existing electromagnetic signal communication is easily intercepted or infiltrated. In order to effectively detect the abnormal electromagnetic signal to find out the specific location, then classify it, it is necessary to study the way of communication. The existing electromagnetic signal detection accuracy is low and cannot be located. Considering the characteristics of different power sources in different locations, combined with spark streaming technology and machine learning classification technology, a joint platform for electromagnetic signal anomaly detection based on big data analysis is proposed. The electromagnetic signal is abnormally detected by feature comparison and small signal analysis, and the position and number between the signal sources are determined by three-point positioning and signal attenuation. The experimental results show that the method can detect abnormal electromagnetic signals and classify abnormal electromagnetic signals well, the accuracy rate can reach 95%, and the positioning accuracy can reach 89%.
44
Meng, Lei. "Internet of Things Information Network Security Situational Awareness Based on Machine Learning Algorithms." Mobile Information Systems 2022 (July 21, 2022): 1–7. http://dx.doi.org/10.1155/2022/4146042.
Full textAbstract:
In order to accurately predict the security situation of Internet of Things information network, a research method based on machine learning algorithm for security situational awareness of Internet of Things information network is proposed. The perception result is represented by the perception model, the sample data are preprocessed based on the linear discriminant analysis method, the sample data are optimized to obtain the combined features, and then the processed data are used as the training input of the RBF neural network to find out the mapping relationship with the network situation value, so as to quantify the security posture of the system. The results show that automatic discovery and classification of seven violations is achieved. Since the platform was launched in China Mobile, more than 65,000 suspected illegal IoT cards have been discovered, effectively monitoring and controlling the operation and behavior of IoT cards. The processing efficiency of IoT card violations has been increased by more than 20 times. After the system is put into use, the discovery of suspected illegal users and behaviors can be realized through full automation, and the process of analysis, confirmation, and disposal can be shortened to 2 hours, which effectively reduce the false alarm rate and reduce operator costs. In previous monitoring of IoT cards, the false-positive rate of illegal IoT cards was about 83%, while the false-positive rate of existing algorithms dropped to 20%.Monitoring shows that business abuse monitoring detects the highest proportion of illegal IoT cards, 59%; infractions of Internet of Things cards detected through Internet abuse monitoring accounted for 20%; the proportion of Internet of Things card with machine card separation is 8%; in the information security risk monitoring (including spam text messages and harassing phone calls), the number of illegal Internet of Things cards found is small, only 3% and 2%, respectively; other infractions, including unauthorized use in locations and user complaints, accounted for 8%. It can effectively improve the ability to discover illegal IoT cards, greatly improve the accuracy of judgment, and improve the efficiency of disposal. The comparison verifies that the method is reliable and effective in the security situation awareness of the Internet of Things information network. Using the Internet of Things information security management system software based on the machine learning algorithm, the system software suitable for anomaly data detection is trained by adjusting the main parameters of the algorithm, which improves the automation and intelligence degree of the system software.
45
Li, Han, Xinyu Wang, Zhongguo Yang, Sikandar Ali, Ning Tong, and Samad Baseer. "Correlation-Based Anomaly Detection Method for Multi-sensor System." Computational Intelligence and Neuroscience 2022 (May 31, 2022): 1–13. http://dx.doi.org/10.1155/2022/4756480.
Full textAbstract:
In industry, sensor-based monitoring of equipment or environment has become a necessity. Instead of using a single sensor, multi-sensor system is used to fully detect abnormalities in complex scenarios. Recently, physical models, signal processing technology, and various machine learning models have improved the performance. However, these methods either do not consider the potential correlation between features or do not take advantage of the sequential changes of correlation while constructing an anomaly detection model. This paper firstly analyzes the correlation characteristic of a multi-sensor system, which shows a lot of clues to the anomaly/fault propagation. Then, a multi-sensor anomaly detection method, which finds and uses the correlation between features contained in the multidimensional time-series data, is proposed. The method converts the multidimensional time-series data into temporal correlation graphs according to time window. By transforming time-series data into graph structure, the task of anomaly detection is considered as a graph classification problem. Moreover, based on the stability and dynamics of the correlation between features, a structure-sensitive graph neural network is used to establish the anomaly detection model, which is used to discover anomalies from multi-sensor system. Experiments on three real-world industrial multi-sensor systems with anomalies indicate that the method obtained better performance than baseline methods, with the mean value of F1 score reaching more than 0.90 and the mean value of AUC score reaching more than 0.95. That is, the method can effectively detect anomalies of multidimensional time series.
46
Wong, Simon, John-Kun-Woon Yeung, Yui-Yip Lau, and Joseph So. "Technical Sustainability of Cloud-Based Blockchain Integrated with Machine Learning for Supply Chain Management." Sustainability 13, no. 15 (July 23, 2021): 8270. http://dx.doi.org/10.3390/su13158270.
Full textAbstract:
Knowing the challenges of keeping and manipulating more and more immutable transaction records in a blockchain network of various supply chain parties and the opportunities of leveraging sophisticated analyses on the big data generated from these records, design of a robust blockchain architecture based on a cloud infrastructure is proposed. This paper presents this technical design with consideration of the technical sustainability in terms of scalability and big data processing and analytics. A case study was used to illustrate how the technical sustainability is achieved by applying the proposed technical design to the real-time detection of the maritime risk management. This case also illustrates how machine learning mechanism helps to reduce maritime risk by guiding a cargo ship to adjust to the planned or safe route from a detour to a danger zone. This paper also discusses the implications for further research direction.
47
Shoukat, Aimen, Muhammad Abul Hassan, Muhammad Rizwan, Muhammad Imad, Farhatullah, Syed Haider Ali, and Sana Ullah. "Design a framework for IoT- Identification, Authentication and Anomaly detection using Deep Learning: A Review." EAI Endorsed Transactions on Smart Cities 7, no. 1 (January 17, 2023): e1. http://dx.doi.org/10.4108/eetsc.v7i1.2067.
Full textAbstract:
The Internet of Things (IoT) connects billions of smart gadgets so that they may communicate with one another without the need for human intervention. With an expected 50 billion devices by the end of 2020, it is one of the fastest-growing industries in computer history. On the one hand, IoT technologies are critical in increasing a variety of real-world smart applications that can help people live better lives. The cross-cutting nature of IoT systems, on the other hand, has presented new security concerns due to the diverse components involved in their deployment. For IoT devices and their inherent weaknesses, security techniques such as encryption, authentication, permissions, network monitoring, \& application security are ineffective. To properly protect the IoT ecosystem, existing security solutions need to be strengthened. Machine learning and deep learning (ML/DL) have come a long way in recent years, and machine intelligence has gone from being a laboratory curiosity to being used in a variety of significant applications. The ability to intelligently monitor IoT devices is an important defense against new or negligible assaults. ML/DL are effective data exploration techniques for learning about 'normal' and 'bad' behavior in IoT devices and systems. Following a comprehensive literature analysis on Machine Learning methods as well as the importance of IoT security within the framework of different sorts of potential attacks, multiple DL algorithms have been evaluated in terms of detecting attacks as well as anomaly detection in this work. We propose a taxonomy of authorization and authentication systems in the Internet of Things based on the review, with a focus on DL-based schemes. The authentication security threats and problems for IoT are thoroughly examined using the taxonomy supplied. This article provides an overview of projects that involve the use of deep learning to efficiently and automatically provide IoT applications.
48
Miller, Andrew, Jan Petrich, and Shashi Phoha. "Advanced Image Analysis for Learning Underlying Partial Differential Equations for Anomaly Identification." Journal of Imaging Science and Technology 64, no. 2 (March 1, 2020): 20510–1. http://dx.doi.org/10.2352/j.imagingsci.technol.2020.64.2.020510.
Full textAbstract:
Abstract In this article, the authors adapt and utilize data-driven advanced image processing and machine learning techniques to identify the underlying dynamics and the model parameters for dynamic processes driven by partial differential equations (PDEs). Potential applications include non-destructive inspection for material crack detection using thermal imaging as well as real-time anomaly detection for process monitoring of three-dimensional printing applications. A neural network (NN) architecture is established that offers sufficient flexibility for spatial and temporal derivatives to capture the physical dependencies inherent in the process. Predictive capabilities are then established by propagating the process forward in time using the acquired model structure as well as individual parameter values. Moreover, deviations in the predicted values can be monitored in real time to detect potential process anomalies or perturbations. For concept development and validation, this article utilizes well-understood PDEs such as the homogeneous heat diffusion equation. Time series data governed by the heat equation representing a parabolic PDE is generated using high-fidelity simulations in order to construct the heat profile. Model structure and parameter identification are realized through a shallow residual convolutional NN. The learned model structure and associated parameters resemble a spatial convolution filter, which can be applied to the current heat profile to predict the diffusion behavior forward in time.
49
Khan, Muhammad Ashfaq, and Juntae Kim. "Toward Developing Efficient Conv-AE-Based Intrusion Detection System Using Heterogeneous Dataset." Electronics 9, no. 11 (October 26, 2020): 1771. http://dx.doi.org/10.3390/electronics9111771.
Full textAbstract:
Recently, due to the rapid development and remarkable result of deep learning (DL) and machine learning (ML) approaches in various domains for several long-standing artificial intelligence (AI) tasks, there has an extreme interest in applying toward network security too. Nowadays, in the information communication technology (ICT) era, the intrusion detection (ID) system has the great potential to be the frontier of security against cyberattacks and plays a vital role in achieving network infrastructure and resources. Conventional ID systems are not strong enough to detect advanced malicious threats. Heterogeneity is one of the important features of big data. Thus, designing an efficient ID system using a heterogeneous dataset is a massive research problem. There are several ID datasets openly existing for more research by the cybersecurity researcher community. However, no existing research has shown a detailed performance evaluation of several ML methods on various publicly available ID datasets. Due to the dynamic nature of malicious attacks with continuously changing attack detection methods, ID datasets are available publicly and are updated systematically. In this research, spark MLlib (machine learning library)-based robust classical ML classifiers for anomaly detection and state of the art DL, such as the convolutional-auto encoder (Conv-AE) for misuse attack, is used to develop an efficient and intelligent ID system to detect and classify unpredictable malicious attacks. To measure the effectiveness of our proposed ID system, we have used several important performance metrics, such as FAR, DR, and accuracy, while experiments are conducted on the publicly existing dataset, specifically the contemporary heterogeneous CSE-CIC-IDS2018 dataset.
50
Anwar, Raja Waseem, Kashif Naseer Qureshi, Wamda Nagmeldin, Abdelzahir Abdelmaboud, Kayhan Zrar Ghafoor, Ibrahim Tariq Javed, and Noel Crespi. "Data Analytics, Self-Organization, and Security Provisioning for Smart Monitoring Systems." Sensors 22, no. 19 (September 22, 2022): 7201. http://dx.doi.org/10.3390/s22197201.
Full textAbstract:
Internet availability and its integration with smart technologies have favored everyday objects and things and offered new areas, such as the Internet of Things (IoT). IoT refers to a concept where smart devices or things are connected and create a network. This new area has suffered from big data handling and security issues. There is a need to design a data analytics model by using new 5G technologies, architecture, and a security model. Reliable data communication in the presence of legitimate nodes is always one of the challenges in these networks. Malicious nodes are generating inaccurate information and breach the user’s security. In this paper, a data analytics model and self-organizing architecture for IoT networks are proposed to understand the different layers of technologies and processes. The proposed model is designed for smart environmental monitoring systems. This paper also proposes a security model based on an authentication, detection, and prediction mechanism for IoT networks. The proposed model enhances security and protects the network from DoS and DDoS attacks. The proposed model evaluates in terms of accuracy, sensitivity, and specificity by using machine learning algorithms.