Relevant bibliographies by topics / Machine learning, big data, anomaly detection, network monitoring

Academic literature on the topic 'Machine learning, big data, anomaly detection, network monitoring'

Author: Grafiati
Published: 14 March 2023

Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles

Select a source type:

Contents

Consult the lists of relevant articles, books, theses, conference reports, and other scholarly sources on the topic 'Machine learning, big data, anomaly detection, network monitoring.'

Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.

You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.

Journal articles on the topic "Machine learning, big data, anomaly detection, network monitoring"

1

Oprea, Simona-Vasilica, Adela Bâra, Florina Camelia Puican, and Ioan Cosmin Radu. "Anomaly Detection with Machine Learning Algorithms and Big Data in Electricity Consumption." Sustainability 13, no. 19 (October 2, 2021): 10963. http://dx.doi.org/10.3390/su131910963.

Full text
Abstract:
When analyzing smart metering data, both reading errors and frauds can be identified. The purpose of this analysis is to alert the utility companies to suspicious consumption behavior that could be further investigated with on-site inspections or other methods. The use of Machine Learning (ML) algorithms to analyze consumption readings can lead to the identification of malfunctions, cyberattacks interrupting measurements, or physical tampering with smart meters. Fraud detection is one of the classical anomaly detection examples, as it is not easy to label consumption or transactional data. Furthermore, frauds differ in nature, and learning is not always possible. In this paper, we analyze large datasets of readings provided by smart meters installed in a trial study in Ireland by applying a hybrid approach. More precisely, we propose an unsupervised ML technique to detect anomalous values in the time series, establish a threshold for the percentage of anomalous readings from the total readings, and then label that time series as suspicious or not. Initially, we propose two types of algorithms for anomaly detection for unlabeled data: Spectral Residual-Convolutional Neural Network (SR-CNN) and an anomaly trained model based on martingales for determining variations in time-series data streams. Then, the Two-Class Boosted Decision Tree and Fisher Linear Discriminant analysis are applied on the previously processed dataset. By training the model, we obtain the required capabilities of detecting suspicious consumers proved by an accuracy of 90%, precision score of 0.875, and F1 score of 0.894.
APA, Harvard, Vancouver, ISO, and other styles
2

Alnafessah, Ahmad, and Giuliano Casale. "Artificial neural networks based techniques for anomaly detection in Apache Spark." Cluster Computing 23, no. 2 (October 23, 2019): 1345–60. http://dx.doi.org/10.1007/s10586-019-02998-y.

Full text
Abstract:
Abstract Late detection and manual resolutions of performance anomalies in Cloud Computing and Big Data systems may lead to performance violations and financial penalties. Motivated by this issue, we propose an artificial neural network based methodology for anomaly detection tailored to the Apache Spark in-memory processing platform. Apache Spark is widely adopted by industry because of its speed and generality, however there is still a shortage of comprehensive performance anomaly detection methods applicable to this platform. We propose an artificial neural networks driven methodology to quickly sift through Spark logs data and operating system monitoring metrics to accurately detect and classify anomalous behaviors based on the Spark resilient distributed dataset characteristics. The proposed method is evaluated against three popular machine learning algorithms, decision trees, nearest neighbor, and support vector machine, as well as against four variants that consider different monitoring datasets. The results prove that our proposed method outperforms other methods, typically achieving 98–99% F-scores, and offering much greater accuracy than alternative techniques to detect both the period in which anomalies occurred and their type.
APA, Harvard, Vancouver, ISO, and other styles
3

Borghesi, Andrea, Andrea Bartolini, Michele Lombardi, Michela Milano, and Luca Benini. "Anomaly Detection Using Autoencoders in High Performance Computing Systems." Proceedings of the AAAI Conference on Artificial Intelligence 33 (July 17, 2019): 9428–33. http://dx.doi.org/10.1609/aaai.v33i01.33019428.

Full text
Abstract:
Anomaly detection in supercomputers is a very difficult problem due to the big scale of the systems and the high number of components. The current state of the art for automated anomaly detection employs Machine Learning methods or statistical regression models in a supervised fashion, meaning that the detection tool is trained to distinguish among a fixed set of behaviour classes (healthy and unhealthy states).We propose a novel approach for anomaly detection in HighPerformance Computing systems based on a Machine (Deep) Learning technique, namely a type of neural network called autoencoder. The key idea is to train a set of autoencoders to learn the normal (healthy) behaviour of the supercomputer nodes and, after training, use them to identify abnormal conditions. This is different from previous approaches which where based on learning the abnormal condition, for which there are much smaller datasets (since it is very hard to identify them to begin with).We test our approach on a real supercomputer equipped with a fine-grained, scalable monitoring infrastructure that can provide large amount of data to characterize the system behaviour. The results are extremely promising: after the training phase to learn the normal system behaviour, our method is capable of detecting anomalies that have never been seen before with a very good accuracy (values ranging between 88% and 96%).
APA, Harvard, Vancouver, ISO, and other styles
4

Albattah, Albatul, and Murad A. Rassam. "A Correlation-Based Anomaly Detection Model for Wireless Body Area Networks Using Convolutional Long Short-Term Memory Neural Network." Sensors 22, no. 5 (March 2, 2022): 1951. http://dx.doi.org/10.3390/s22051951.

Full text
Abstract:
As the Internet of Healthcare Things (IoHT) concept emerges today, Wireless Body Area Networks (WBAN) constitute one of the most prominent technologies for improving healthcare services. WBANs are made up of tiny devices that can effectively enhance patient quality of life by collecting and monitoring physiological data and sending it to healthcare givers to assess the criticality of a patient and act accordingly. The collected data must be reliable and correct, and represent the real context to facilitate right and prompt decisions by healthcare personnel. Anomaly detection becomes a field of interest to ensure the reliability of collected data by detecting malicious data patterns that result due to various reasons such as sensor faults, error readings and possible malicious activities. Various anomaly detection solutions have been proposed for WBAN. However, existing detection approaches, which are mostly based on statistical and machine learning techniques, become ineffective in dealing with big data streams and novel context anomalous patterns in WBAN. Therefore, this paper proposed a model that employs the correlations that exist in the different physiological data attributes with the ability of the hybrid Convolutional Long Short-Term Memory (ConvLSTM) techniques to detect both simple point anomalies as well as contextual anomalies in the big data stream of WBAN. Experimental evaluations revealed that an average of 98% of F1-measure and 99% accuracy were reported by the proposed model on different subjects of the datasets compared to 64% achieved by both CNN and LSTM separately.
APA, Harvard, Vancouver, ISO, and other styles
5

Chen, Naiyue, Yi Jin, Yinglong Li, and Luxin Cai. "Trust-based federated learning for network anomaly detection." Web Intelligence 19, no. 4 (January 20, 2022): 317–27. http://dx.doi.org/10.3233/web-210475.

Full text
Abstract:
With the rapid development of social networks and the massive popularity of intelligent mobile terminals, network anomaly detection is becoming increasingly important. In daily work and life, edge nodes store a large number of network local connection data and audit data, which can be used to analyze network abnormal behavior. With the increasingly close network communication, the amount of network connection and other related data collected by each network terminal is increasing. Machine learning has become a classification method to analyze the features of big data in the network. Face to the problems of excessive data and long response time for network anomaly detection, we propose a trust-based Federated learning anomaly detection algorithm. We use the edge nodes to train the local data model, and upload the machine learning parameters to the central node. Meanwhile, according to the performance of edge nodes training, we set different weights to match the processing capacity of each terminal which will obtain faster convergence speed and better attack classification accuracy. The user’s private information will only be processed locally and will not be uploaded to the central server, which can reduce the risk of information disclosure. Finally, we compare the basic federated learning model and TFCNN algorithm on KDD Cup 99 dataset and MNIST dataset. The experimental results show that the TFCNN algorithm can improve accuracy and communication efficiency.
APA, Harvard, Vancouver, ISO, and other styles
6

Do, ChoXuan, Nguyen Quang Dam, and Nguyen Tung Lam. "Optimization of network traffic anomaly detection using machine learning." International Journal of Electrical and Computer Engineering (IJECE) 11, no. 3 (June 1, 2021): 2360. http://dx.doi.org/10.11591/ijece.v11i3.pp2360-2370.

Full text
Abstract:
In this paper, to optimize the process of detecting cyber-attacks, we choose to propose 2 main optimization solutions: Optimizing the detection method and optimizing features. Both of these two optimization solutions are to ensure the aim is to increase accuracy and reduce the time for analysis and detection. Accordingly, for the detection method, we recommend using the Random Forest supervised classification algorithm. The experimental results in section 4.1 have proven that our proposal that use the Random Forest algorithm for abnormal behavior detection is completely correct because the results of this algorithm are much better than some other detection algorithms on all measures. For the feature optimization solution, we propose to use some data dimensional reduction techniques such as information gain, principal component analysis, and correlation coefficient method. The results of the research proposed in our paper have proven that to optimize the cyber-attack detection process, it is not necessary to use advanced algorithms with complex and cumbersome computational requirements, it must depend on the monitoring data for selecting the reasonable feature extraction and optimization algorithm as well as the appropriate attack classification and detection algorithms.
APA, Harvard, Vancouver, ISO, and other styles
7

Vajda, Daniel, Adrian Pekar, and Karoly Farkas. "Towards Machine Learning-based Anomaly Detection on Time-Series Data." Infocommunications journal 13, no. 1 (2021): 35–44. http://dx.doi.org/10.36244/icj.2021.1.5.

Full text
Abstract:
The complexity of network infrastructures is exponentially growing. Real-time monitoring of these infrastructures is essential to secure their reliable operation. The concept of telemetry has been introduced in recent years to foster this process by streaming time-series data that contain feature-rich information concerning the state of network components. In this paper, we focus on a particular application of telemetry — anomaly detection on time-series data. We rigorously examined state-of-the-art anomaly detection methods. Upon close inspection of the methods, we observed that none of them suits our requirements as they typically face several limitations when applied on time-series data. This paper presents Alter-Re2, an improved version of ReRe, a state-of-the-art Long Short- Term Memory-based machine learning algorithm. Throughout a systematic examination, we demonstrate that by introducing the concepts of ageing and sliding window, the major limitations of ReRe can be overcome. We assessed the efficacy of Alter-Re2 using ten different datasets and achieved promising results. Alter-Re2 performs three times better on average when compared to ReRe.
APA, Harvard, Vancouver, ISO, and other styles
8

Novoa-Paradela, David, Óscar Fontenla-Romero, and Bertha Guijarro-Berdiñas. "Adaptive Real-Time Method for Anomaly Detection Using Machine Learning." Proceedings 54, no. 1 (August 22, 2020): 38. http://dx.doi.org/10.3390/proceedings2020054038.

Full text
Abstract:
Anomaly detection is a sub-area of machine learning that deals with the development of methods to distinguish among normal and anomalous data. Due to the frequent use of anomaly-detection systems in monitoring and the lack of methods capable of learning in real time, this research presents a new method that provides such online adaptability. The method bases its operation on the properties of scaled convex hulls. It begins building a convex hull, using a minimum set of data, that is adapted and subdivided along time to accurately fit the boundary of the normal class data. The model has online learning ability and its execution can be carried out in a distributed and parallel way, all of them interesting advantages when dealing with big datasets. The method has been compared to other state-of-the-art algorithms demonstrating its effectiveness.
APA, Harvard, Vancouver, ISO, and other styles
9

Chimphlee, Siriporn, and Witcha Chimphlee. "Machine learning to improve the performance of anomaly-based network intrusion detection in big data." Indonesian Journal of Electrical Engineering and Computer Science 30, no. 2 (May 1, 2023): 1106. http://dx.doi.org/10.11591/ijeecs.v30.i2.pp1106-1119.

Full text
Abstract:
With the rapid growth of digital technology communications are overwhelmed by network data traffic. The demand for the internet is growing every day in today's cyber world, raising concerns about network security. Big Data are a term that describes a vast volume of complicated data that is critical for evaluating network patterns and determining what has occurred in the network. Therefore, detecting attacks in a large network is challenging. Intrusion detection system (IDS) is a promising cybersecurity research field. In this paper, we proposed an efficient classification scheme for IDS, which is divided into two procedures, on the CSE-CIC-IDS-2018 dataset, data pre-processing techniques including under-sampling, feature selection, and classifier algorithms were used to assess and decide the best performing model to classify invaders. We have implemented and compared seven classifier machine learning algorithms with various criteria. This work explored the application of the random forest (RF) for feature selection in conjunction with machine learning (ML) techniques including linear regression (LR), k-Nearest Neighbor (k-NN), classification and regression trees (CART), Bayes, RF, multi layer perceptron (MLP), and XGBoost in order to implement IDSS. The experimental results show that the MLP algorithm in the most successful with best performance with evaluation matrix.
APA, Harvard, Vancouver, ISO, and other styles
10

Káš, M., and F. F. Wamba. "Anomaly detection-based condition monitoring." Insight - Non-Destructive Testing and Condition Monitoring 64, no. 8 (August 1, 2022): 453–58. http://dx.doi.org/10.1784/insi.2022.64.8.453.

Full text
Abstract:
The impact of an anomaly is domain-dependent. In a dataset of network activities, an anomaly can imply an intrusion attack. Other objectives of anomaly detection are industrial damage detection, data leak prevention, identifying security vulnerabilities or military surveillance. Anomalies are observations or a sequence of observations in which the distribution deviates remarkably from the general distribution of the whole dataset. A large majority of the dataset consists of normal (healthy) data points. The anomalies form only a very small part of the dataset. Anomaly detection is the technique used to find these observations and its methods are specific to the type of data. While there is a wide spectrum of anomaly detection approaches available today, it becomes increasingly difficult to keep track of all the techniques. In fact, it is not clear which of the three categories of detection methods, ie statistical approaches, machine learning approaches or deep learning approaches, is more appropriate to detect anomalies in time-series data, which are mainly used in industry. A typical industrial device has multi-dimensional characteristics. It is possible to measure voltage, current, active power, vibrations, rotational speed, temperature, pressure difference, etc, on such a device. Early detection of the anomalous behaviour of industrial devices can help reduce or prevent serious damage, which could lead to significant financial loss. This paper presents a summary of the methods used to detect anomalies in condition monitoring applications.
APA, Harvard, Vancouver, ISO, and other styles
More sources

Dissertations / Theses on the topic "Machine learning, big data, anomaly detection, network monitoring"

1

Syal, Astha. "Automatic Network Traffic Anomaly Detection and Analysis using SupervisedMachine Learning Techniques." Youngstown State University / OhioLINK, 2019. http://rave.ohiolink.edu/etdc/view?acc_num=ysu1578259840945109.

Full text
APA, Harvard, Vancouver, ISO, and other styles
2

Jehangiri, Ali Imran. "Distributed Anomaly Detection and Prevention for Virtual Platforms." Doctoral thesis, 2015. http://hdl.handle.net/11858/00-1735-0000-0022-605F-2.

Full text
APA, Harvard, Vancouver, ISO, and other styles
3

(10723926), Adefolarin Alaba Bolaji. "Community Detection of Anomaly in Large-Scale Network Dissertation - Adefolarin Bolaji .pdf." Thesis, 2021.

Find full text
Abstract:

The detection of anomalies in real-world networks is applicable in different domains; the application includes, but is not limited to, credit card fraud detection, malware identification and classification, cancer detection from diagnostic reports, abnormal traffic detection, identification of fake media posts, and the like. Many ongoing and current researches are providing tools for analyzing labeled and unlabeled data; however, the challenges of finding anomalies and patterns in large-scale datasets still exist because of rapid changes in the threat landscape.

In this study, I implemented a novel and robust solution that combines data science and cybersecurity to solve complex network security problems. I used Long Short-Term Memory (LSTM) model, Louvain algorithm, and PageRank algorithm to identify and group anomalies in large-scale real-world networks. The network has billions of packets. The developed model used different visualization techniques to provide further insight into how the anomalies in the network are related.

Mean absolute error (MAE) and root mean square error (RMSE) was used to validate the anomaly detection models, the results obtained for both are 5.1813e-04 and 1e-03 respectively. The low loss from the training phase confirmed the low RMSE at loss: 5.1812e-04, mean absolute error: 5.1813e-04, validation loss: 3.9858e-04, validation mean absolute error: 3.9858e-04. The result from the community detection shows an overall modularity value of 0.914 which is proof of the existence of very strong communities among the anomalies. The largest sub-community of the anomalies connects 10.42% of the total nodes of the anomalies.

The broader aim and impact of this study was to provide sophisticated, AI-assisted countermeasures to cyber-threats in large-scale networks. To close the existing gaps created by the shortage of skilled and experienced cybersecurity specialists and analysts in the cybersecurity field, solutions based on out-of-the-box thinking are inevitable; this research was aimed at yielding one of such solutions. It was built to detect specific and collaborating threat actors in large networks and to help speed up how the activities of anomalies in any given large-scale network can be curtailed in time.


APA, Harvard, Vancouver, ISO, and other styles

Book chapters on the topic "Machine learning, big data, anomaly detection, network monitoring"

1

Kumar, Shailender, Namrata Jha, and Nikhil Sachdeva. "A Deep Learning Approach for Anomaly-Based Network Intrusion Detection Systems: A Survey and an Objective Comparison." In Machine Learning and Big Data Analytics (Proceedings of International Conference on Machine Learning and Big Data Analytics (ICMLBDA) 2021), 227–35. Cham: Springer International Publishing, 2021. http://dx.doi.org/10.1007/978-3-030-82469-3_20.

Full text
APA, Harvard, Vancouver, ISO, and other styles
2

Narayan, Valliammal, and Shanmugapriya D. "Big Data Analytics With Machine Learning and Deep Learning Methods for Detection of Anomalies in Network Traffic." In Research Anthology on Big Data Analytics, Architectures, and Applications, 678–707. IGI Global, 2022. http://dx.doi.org/10.4018/978-1-6684-3662-2.ch032.

Full text
Abstract:
Information is vital for any organization to communicate through any network. The growth of internet utilization and the web users increased the cyber threats. Cyber-attacks in the network change the traffic flow of each system. Anomaly detection techniques have been developed for different types of cyber-attack or anomaly strategies. Conventional ADS protect information transferred through the network or cyber attackers. The stable prevention of anomalies by machine and deep-learning algorithms are applied for cyber-security. Big data solutions handle voluminous data in a short span of time. Big data management is the organization and manipulation of huge volumes of structured data, semi-structured data and unstructured data, but it does not handle a data imbalance problem during the training process. Big data-based machine and deep-learning algorithms for anomaly detection involve the classification of decision boundary between normal traffic flow and anomaly traffic flow. The performance of anomaly detection is efficiently increased by different algorithms.
APA, Harvard, Vancouver, ISO, and other styles
3

Narayan, Valliammal, and Shanmugapriya D. "Big Data Analytics With Machine Learning and Deep Learning Methods for Detection of Anomalies in Network Traffic." In Handbook of Research on Machine and Deep Learning Applications for Cyber Security, 317–46. IGI Global, 2020. http://dx.doi.org/10.4018/978-1-5225-9611-0.ch015.

Full text
Abstract:
Information is vital for any organization to communicate through any network. The growth of internet utilization and the web users increased the cyber threats. Cyber-attacks in the network change the traffic flow of each system. Anomaly detection techniques have been developed for different types of cyber-attack or anomaly strategies. Conventional ADS protect information transferred through the network or cyber attackers. The stable prevention of anomalies by machine and deep-learning algorithms are applied for cyber-security. Big data solutions handle voluminous data in a short span of time. Big data management is the organization and manipulation of huge volumes of structured data, semi-structured data and unstructured data, but it does not handle a data imbalance problem during the training process. Big data-based machine and deep-learning algorithms for anomaly detection involve the classification of decision boundary between normal traffic flow and anomaly traffic flow. The performance of anomaly detection is efficiently increased by different algorithms.
APA, Harvard, Vancouver, ISO, and other styles
4

Vidal, Jorge Maestre, Marco Antonio Sotelo Monge, and Sergio Mauricio Martínez Monterrubio. "Anomaly-Based Intrusion Detection." In Handbook of Research on Machine and Deep Learning Applications for Cyber Security, 195–218. IGI Global, 2020. http://dx.doi.org/10.4018/978-1-5225-9611-0.ch010.

Full text
Abstract:
Anomaly-based intrusion detection has become an indispensable player on the existing cybersecurity landscape, where it enables the identification of suspicious behaviors that significantly differ from normal activities. In this way, it is possible to discover never-seen-before threats and provide zero-day recognition capabilities. But the recent advances on communication technologies are leading to changes in the monitoring scenarios that result in novel challenges to be taken into consideration, as is the case of greater data heterogeneity, adversarial attacks, energy consumption, or lack of up-to-date datasets. With the aim on bringing the reader closer to them, this chapter deepens the following topics: evolution of the anomaly definition, anomaly recognition for network-based intrusion detection, outlier characterizations, knowledge acquisition for usage modelling, distances and similarity measures for decision-making, anomaly recognition and non-stationarity, metrics and evaluation methodologies, and challenges related with the emergent monitorization environments.
APA, Harvard, Vancouver, ISO, and other styles
5

Zhao, Peng, Yuan Ren, and Xi Chen. "Big Data Helps for Non-Pharmacological Disease Control Measures of COVID-19." In Encyclopedia of Data Science and Machine Learning, 143–55. IGI Global, 2022. http://dx.doi.org/10.4018/978-1-7998-9220-5.ch009.

Full text
Abstract:
This article reveals how artificial intelligence and big data analytics help the non-pharmacological disease control measures. Several cutting-edge technologies are illustrated in terms of the system architecture, the data workflows, and the machine learning/deep learning models. This article will also investigate a comprehensive social control system that is designed for disease control measures by integrating the above mentioned technologies. For each component of the system, real-world applications will be represented in the form of examining the capability of the proposed models. The proposed system can detect whether people are keeping social distancing and wearing a facial mask in public spaces, along with measuring the mobility assessment, which can be applied to screen the stay-at-home orders using big data and visual mining. A fine-tuned CNN-based network will be applied for monitoring the social distancing, while the face mask detection module is trained by fine-tuning the MobileNet architecture.
APA, Harvard, Vancouver, ISO, and other styles

Conference papers on the topic "Machine learning, big data, anomaly detection, network monitoring"

1

Nivlet, Philippe, Knut Steinar Bjorkevoll, Mandar Tabib, Jan Ole Skogestad, Bjornar Lund, Roar Nybo, and Adil Rasheed. "Towards Real-Time Bad Hole Cleaning Problem Detection Through Adaptive Deep Learning Models." In Middle East Oil, Gas and Geosciences Show. SPE, 2023. http://dx.doi.org/10.2118/213643-ms.

Full text
Abstract:
Abstract Monitoring of Equivalent Circulating Density (ECD) may improve assessment of potential bad hole cleaning conditions if calculated and measured sufficiently accurately. Machine learning (ML) models can be used for predicting ECD integrating both along-string and surface drilling measurements and physics-based model (PBM) results, even though their generalization is often challenging. To remediate this generalizability issue, we present an adaptative predictive deep-learning model that is retrained with new measurements in real-time, conditionally that the new measurements are not detected as anomalies. Past ECD measurements, corresponding values predicted by a 1D PBM and other drilling measurements are used as input to a deep learning model, which is pretrained on historical drilling data without any hole cleaning problem. This model has two components: an anomaly detector, and a predictor. In this paper, both components are based on combinations of Long Short-Term Memory (LSTM) cells that allow (1) to account for data correlations between the different time series and between the different time stamps, and (2) generate future data conditioned to past observations. As drilling progresses, new data is proposed to the anomaly detector: if the network fails to reconstruct them correctly, an alarm is raised. Otherwise, the new data is used to retrain the models. We show the benefits of such an approach on two real examples from offshore Norway with increasing complexity: For the first one, with no major drilling issue, we simply use ECD from the PBM to predict ECD ahead of the bit. The second example had multiple issues linked with mud loss and poor hole cleaning. For this latter case, we used additional topside measurements to better constrain the ECD prediction.
APA, Harvard, Vancouver, ISO, and other styles
2

Liu, Zhipeng, Niraj Thapa, Addison Shaver, Kaushik Roy, Xiaohong Yuan, and Sajad Khorsandroo. "Anomaly Detection on IoT Network Intrusion Using Machine Learning." In 2020 International Conference on Artificial Intelligence, Big Data, Computing and Data Communication Systems (icABCD). IEEE, 2020. http://dx.doi.org/10.1109/icabcd49160.2020.9183842.

Full text
APA, Harvard, Vancouver, ISO, and other styles
3

Wang, Biao, Guoqing Han, Xin Lu, Shuai Tan, Zhiyong Zhu, and Huizhu Xiang. "Remote Monitoring of Well Production Performance Based on Machine Learning." In SPE Western Regional Meeting. SPE, 2022. http://dx.doi.org/10.2118/209255-ms.

Full text
Abstract:
Abstract A set of dynamic remote monitoring method of production performance based on Machine Learning is proposed for the production process of electric submersible pump (ESP) well with multi-dimensional parameters. Aiming at dealing with the characteristics of multi-dimensional parameters in the complex production system, to implement dynamic monitoring of production performance. Helping the engineers at data centers to find the anomaly remotely and make response in a timely manner. It puts forward a procedure for large amount, high dimension and low information density production data in complex production system, using the dimensionality reduction algorithm to reduce the dimensionality into one comprehensive parameter changing over time, time series analysis algorithm for the production anomaly detection and prediction based on Machine Learning. The Principal Component Analysis (PCA) is used to reduce the dimensionality and extract the crucial information. The Autoregressive Integrated Moving Average (ARIMA) model is used to conduct timing anomaly detection, and fbProphet model is used to analyze the dimensionality reduced data to provide prediction of the production. With the dimensionality reduction, time series comprehensive parameter analysis and anomaly detection method based on Machine Learning, more than 40 ESP wells with 15 dimensions production daily parameters up to 1,000 days were analyzed, which realized the comprehensive description of ESP wells with multiparameter. Although the PCA retained only 47.73% of the information in the first principal component, which may be related with the low information density of industrial big data, the subsequent analysis proved the effectiveness. The time series analysis realized many times anomaly detection during the life period of each ESP well, and visualized the production data and the anomalous events. More than 100 anomalous events were detected in advance and which were robust corresponding to the subsequence real production events, among which 95% agreement rate is achieved. The procedure proposed reported the anomaly events with high confidence up to 90%, and low misstatement rate and omission rate, realized the production perception and abnormal detection in a timely manner. Based on this algorithm, the best time for the well intervention is determined, so that the loss of production is avoided and the revenue is maximized. The novelty of the procedure of Machine Learning using the multiple production data is in the ability to provide a solution of dealing with the low information density and high noise in the complex multi- dimensional production data of production wells, realize the comprehensive description, analysis and prediction of the production. It is helpful for engineers find the abnormalities in time, and will support the decision making of production, optimization and well intervention for the production.
APA, Harvard, Vancouver, ISO, and other styles
4

Cadei, Luca, Gianmarco Rossi, Lorenzo Lancia, Danilo Loffreno, Andrea Corneo, Diletta Milana, Marco Montini, et al. "Hazardous Events Prevention and Management Through an Integrated Machine Learning and Big Data Analytics Framework." In SPE Conference at Oman Petroleum & Energy Show. SPE, 2022. http://dx.doi.org/10.2118/200110-ms.

Full text
Abstract:
Abstract This paper reports the development and tests of an advance methodologies to predict Upstream plant risky events, such as flaring, applying an integrated framework. The core idea is to exploit Machine Learning and big data analytics techniques to tackle and manage both major upsets that would lead to significant inefficiency and loss. The tool is developed for complex upstream production system, where upset could be caused by a huge amount of heterogeneous factors, exploiting data driven monitoring systems to identify the weak signals of the upcoming events. The framework proposed is mainly composed by a strong pipeline divided in 3 modules operating before (predictive phase), during and after the event. The former aims to reduce the probability of an event, the second works on the severity and the third one has a dual function: reporting upsets and feedback gathering system to be used to further improve the analytics implemented. The Predictive component alerts operators when it recognizes a dangerous pattern among the parameters considered. The other two components can support this one and can be exploited to detect early signs of deviations from the proper operating envelope, while predictive performances are not satisfying. Moreover, during an event occurrence, operators can promptly identify the causes of the upset through the entire production system. This allows a faster reaction and consequently a significant reduction in magnitude. The solution proposed provides 2 complementary methodologies: an agnostic anomaly detection system, helping to map plant functional unit anomalous behavior, as a dynamic operating envelope, and identifying the most affected ones; A real time root-cause analysis, as a vertical solution, obtained learning from the monitoring of the different specific functional unit; The tool is also able to provide an automatic event register using information provided by the root-cause system, including operator feedbacks that will improve the performances of each module of the framework. The entire pipeline developed has been applied on-line, working with real time data coming from an operating oilfield, with special focus on blowdown and flaring system. The robust architecture generated is able to overcome some main issues related to the complexity of Upstream production assets such as lack of data, quick dynamic of physical phenomena analysed and randomness of upsets. The first test demonstrates that the tool accuracy allows to identify and suggest actions on 35% of the most dangerous flaring events occurring. Moreover, the effectiveness increase significantly proving a real time root-cause analysis considering both strong and weak signals that cause dangerous overpressures through the treatment plant.
APA, Harvard, Vancouver, ISO, and other styles
5

Peng, Dandan, Chenyu Liu, Wim Desmet, and Konstantinos Gryllias. "Condition Monitoring of Wind Turbines Based on Anomaly Detection Using Deep Support Vector Data Description." In ASME Turbo Expo 2022: Turbomachinery Technical Conference and Exposition. American Society of Mechanical Engineers, 2022. http://dx.doi.org/10.1115/gt2022-82624.

Full text
Abstract:
Abstract Wind turbine condition monitoring is considered as a key task in wind power industry. A plethora of methodologies based on machine learning have been proposed but the absence of faulty data, at the amount and the variety needed, still set limitations. Therefore Anomaly Detection methodologies are proposed as alternatives for fault detection. Deep learning tools have been introduced in the research field of wind turbines’ monitoring for the purpose of higher detection accuracy. In this work, a deep learning-based anomaly detection method, the Deep Support Vector Data Description (Deep SVDD), is proposed for the monitoring of wind turbines. Compared to the classic SVDD anomaly detection approach, this method combines a deep network, more specifically a Convolutional Neural Network (CNN), with the SVDD detector in order to automatically extract effective features. To test and validate the effectiveness of the proposed method, the Deep SVDD method is applied on SCADA data from a real wind turbine use case, targeting to the ice detection on wind turbine blades. The experimental results show that the method can effectively detect the generation of ice on wind turbines’ blades with a successful detection rate of 91.45%.
APA, Harvard, Vancouver, ISO, and other styles
6

Ahmed, Muhammad Shahzad, Mahdi Abdula Al Bloushi, and Asad Ali. "Case Study: Application of Wireless Condition Based Monitoring by Applying Machine Learning Models." In ADIPEC. SPE, 2022. http://dx.doi.org/10.2118/211258-ms.

Full text
Abstract:
Abstract In oil and gas and petrochemical industries, high and medium critical rotating equipment e.g., Compressors, Pumps etc. are normally equipped with fixed machine vibration monitoring systems for the online monitoring and protection of the equipment. However, there are multiple low and medium critical and legacy equipment which are without fixed vibration monitoring system and where installing fixed vibration system is not financially practical. However, looking at the scale of a cost-efficient machine monitoring system and changing the maintenance approach from Preventive to Predictive can have significant financial as well as operational benefit. The intent behind performing this study was to assess the utilization of wireless vibration monitoring with following applications in One of Giant Oil and Gas Production Field (X Field) in ADNOC Onshore. Rotating equipment with failure history e.g., Instrument Air compressors, HVAC Compressors etc. Multiphase Pumps with no fixed vibration monitoring system. Centralized monitoring of remote equipment by deploying Private LPWA (Low Power Wide Area) Network over existing Telecom Backhaul consisting of Fiber Optics and WiMAX wireless networks. In the pursuit of some wireless vibration sensors with long range wireless coverage and on-premises monitoring and analytics application, ADNOC team identified a newly developed solution by renowned industrial instrumentation OEM. This system includes wireless LoRaWAN vibration sensors along with anomaly detection system based on the data collected by wireless sensors. To evaluate the system comparative effectiveness, a Proof of Concept was carried out in X Field by installing the solution at a pump already equipped with fixed vibration monitoring system at a remote facility 25 Km away from Central Plant. The data from these sensors was wirelessly transmitted to LoRaWAN gateway installed 200 meters away from the pump. From the gateway the data was routed to on-premises application server installed in a Central Facility utilizing existing Telecom Backhaul. A decision-based application was used for monitoring, trending, and automatic anomaly detection. At first stage the system was kept at machine learning phase to allow the system to learn the normal behavior of the Pump. Based on this learning data, an AI (Artificial Intelligence) based model was developed which self-assign a decision threshold for anomaly detection and alarming. This solution based on LPWAN (Lowe Power Wide Area Network) technology, LoRaWAN, can be utilized for Condition Based Monitoring, Trending and Anomaly detection of low and medium critical rotating equipment, where installation of fixed vibration monitoring system is not feasible. LoRaWAN sensors provide reliable wireless link up to 1 Km in congested plant installations with no requirement of Line of Sight. One LoRaWAN Gateway can support up to 1,000 Sensors. A cost comparison was also performed with traditional wired and this wireless solution and the later was found to be more cost effective with simplicity in deployment and no major footprints. Most of the available LPWA (LoRaWAN) solution are based on 3rd party connectivity e.g., GSM/Satellite and Cloud based Application Servers. Private LPWAN built on existing SCADA/Telecom infrastructure and on-premises application/network servers are best suited for the application where complete ownership of network/data is required.
APA, Harvard, Vancouver, ISO, and other styles
7

Andreoni Lopez, Martin E., Otto Carlos Muniz Bandeira Duarte, and Guy Pujolle. "A Monitoring and Threat Detection System Using Stream Processing as a Virtual Function for Big Data." In XXXVII Simpósio Brasileiro de Redes de Computadores e Sistemas Distribuídos. Sociedade Brasileira de Computação - SBC, 2019. http://dx.doi.org/10.5753/sbrc_estendido.2019.7789.

Full text
Abstract:
The late detection of security threats causes a significant increase in the risk of irreparable damages, disabling any defense attempt. As a consequence, fast real-time threat detection is mandatory for security guarantees. In addition, Network Function Virtualization (NFV) provides new opportunities for efficient and low-cost security solutions. We propose a fast and efficient threat detection system based on stream processing and machine learning algorithms. The main contributions of this work are i) a novel monitoring threat detection system based on stream processing; ii) two datasets, first a dataset of synthetic security data containing both legitimate and malicious traffic, and the second, a week of real traffic of a telecommunications operator in Rio de Janeiro, Brazil; iii) a data pre-processing algorithm, a normalizing algorithm and an algorithm for fast feature selection based on the correlation between variables; iv) a virtualized network function in an open-source platform for providing a real-time threat detection service; v) near-optimal placement of sensors through a proposed heuristic for strategically positioning sensors in the network infrastructure, with a minimum number of sensors; and, finally, vi) a greedy algorithm that allocates on demand a sequence of virtual network functions.
APA, Harvard, Vancouver, ISO, and other styles
8

Harputlu Aksu, Şeniz, and Erman Çakıt. "Classifying mental workload using EEG data: A machine learning approach." In 13th International Conference on Applied Human Factors and Ergonomics (AHFE 2022). AHFE International, 2022. http://dx.doi.org/10.54941/ahfe1001820.

Full text
Abstract:
Mental workload is related to the difference between the available mental resource capacity of the operator and the mental resource required by the job. To decide the number of tasks assigned to operator and the difficulty levels of those tasks, it is important to know the operator's mental workload. An overload occurs if the amount of resources required by the task exceeds the available capacity of the person. Mental workload analysis helps to recognize the mental fatigue, evaluate the human performance of different level tasks and adjust cognitive sources for safe and efficient human-machine interactions. Excessive levels of mental workload can lead to errors or delays in information processing. Monitoring brain activity has been verified to be sensitive and consistent reflector of mental workload changes. Classification, regression, clustering, anomaly detection, dimensionality reduction, and reward maximization are common machine learning models. Classification of mental workload has critical importance in the domain of human factors and ergonomics. In recent years, with the need to analyze continuous and large-scale data obtained by physiological methods, the use of machine learning algorithms has become widespread in estimating and classifying mental workload. The objectives of the current study were two-fold: (1) to investigate the relationship among EEG features, task difficulty levels and subjective self-assessment (NASA-TLX) scores and (2) to develop machine learning algorithms for classifying mental workload using EEG features. N-back tasks have been commonly used in the literature. In this study, N-back memory tests were performed at four different difficulty levels. As the number of n increases, so does the difficulty of the task. Four participants performed the tests. Seventy EEG features (5 frequency band power for 14 channels) were selected as independent variables. One output variable reflecting the difficulty level of N-Back memory was classified. The machine learning algorithms used in our study were K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Artificial Neural Network (ANN), Random Forest (RF), Gradient Boosting Machine (GBM), Light Gradient Boosting Machine (LightGBM) and Extreme Gradient Boosting (XGBoost) algorithms. As the task difficulty increased, theta activity in prefrontal and frontal regions increased. Especially frontal theta power, parietal and occipital gamma power were significantly correlated to perceived workload scores obtained via NASA-TLX. Prefrontal beta-high activity had a significant negative relationship with self-assessment workload ratings. Prefrontal and frontal theta, prefrontal beta-high, occipital, parietal and temporal gamma and occipital alpha activities were found to be the most effective parameters. The results obtained for the four classes of classification problem reached the accuracy of 68% with EEG features as input and the Random Forest algorithm. In addition, the results obtained for the two classes of classification problem reached the accuracy of 87% with EEG features as input and the GBM algorithm. The results from the analysis indicate that EEG signals play an important role in the classification of mental workload. Another remarkable result was high classification performance of GBM, LightGBM and XGBoost algorithms that have been developed in the recent past and therefore not frequently used in studies on this subject in the literature.
APA, Harvard, Vancouver, ISO, and other styles
9

Buiu, Catalin, and Vladrares Danaila. "DATA SCIENCE AND MACHINE LEARNING TECHNIQUES FOR CASE-BASED LEARNING IN MEDICAL BIOENGINEERING EDUCATION." In eLSE 2020. University Publishing House, 2020. http://dx.doi.org/10.12753/2066-026x-20-194.

Full text
Abstract:
Data science and artificial intelligence (AI) are the main factors driving a new technological revolution. Just recently (November 2019), key U.S. policymakers have announced intentions to create an agency that would invest $100 billion over 5 years on basic research in AI, with a focus on quantum computing, robotics, cybersecurity, and synthetic biology. The need for well educated people in these areas is growing exponentially, and this is more stringent than ever for medical bioengineering professionals who are expected to play a leading role in the promotion of advanced algorithms and methods to advance health care in fields like diagnosis, monitoring, and therapy. In a recent study on the current research areas of big data analytics and AI in health care, the authors have performed a systematic review of literature and found that out the primary interest area proved to be medical image processing and analysis (587 entries out of 2421 articles analysed) followed by decision-support systems and text mining and analysis. Case-based learning is an instructional design model that is learner-centered and intensively used across a variety of disciplines. In this paper, we present a set of tools and a case study that would help medical bioengineering students to grasp both theoretical concepts (both medical, such as gynecological disorders and technological, such as deep learning, neural network architectures, learning algorithms) and delve into practical applications of these techniques in medical image processing. The case study concerns the automated diagnosis of cervigrams (also called cervicographic images), that are colposcopy images used by the gynecologist for cervical cancer diagnosis, study and training. The tools described in this paper are based on using PyTorch, Keras and Tensor Flow. They allow image segmentation, automated detection of cervix, and cervical cancer classification, while also sustaining an intense interaction between participants to the case study. Based on these tools (for which we describe their distinctive advantages and provide comparisons in terms of accuracy and speed), we describe in full details different teaching strategies.
APA, Harvard, Vancouver, ISO, and other styles
You might also be interested in the extended bibliographies on the topic 'Machine learning, big data, anomaly detection, network monitoring' for particular source types:
Journal articles
We offer discounts on all premium plans for authors whose works are included in thematic literature selections. Contact us to get a unique promo code!

To the bibliography