Academic literature on the topic 'Data imbalance problem'
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the lists of relevant articles, books, theses, conference reports, and other scholarly sources on the topic 'Data imbalance problem.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Journal articles on the topic "Data imbalance problem"
1
Tiwari, Himani. "Improvising Balancing Methods for Classifying Imbalanced Data." International Journal for Research in Applied Science and Engineering Technology 9, no. 9 (September 30, 2021): 1535–43. http://dx.doi.org/10.22214/ijraset.2021.38225.
Full textAbstract:
Abstract: Class Imbalance problem is one of the most challenging problems faced by the machine learning community. As we refer the imbalance to various instances in class of being relatively low as compare to other data. A number of over - sampling and under-sampling approaches have been applied in an attempt to balance the classes. This study provides an overview of the issue of class imbalance and attempts to examine various balancing methods for dealing with this problem. In order to illustrate the differences, an experiment is conducted using multiple simulated data sets for comparing the performance of these oversampling methods on different classifiers based on various evaluation criteria. In addition, the effect of different parameters, such as number of features and imbalance ratio, on the classifier performance is also evaluated. Keywords: Imbalanced learning, Over-sampling methods, Under-sampling methods, Classifier performances, Evaluationmetrices
2
Isabella, S. Josephine, Sujatha Srinivasan, and G. Suseendran. "A Framework Using Binary Cross Entropy - Gradient Boost Hybrid Ensemble Classifier for Imbalanced Data Classification." Webology 18, no. 1 (April 29, 2021): 104–20. http://dx.doi.org/10.14704/web/v18i1/web18076.
Full textAbstract:
During the big data era, there is a continuous occurrence of developing the learning of imbalanced data gives a pathway for the research field along with data mining and machine learning concepts. In recent years, Big Data and Big Data Analytics having high eminence due to data exploration by many of the applications in real-time. Using machine learning will be a greater solution to solve the difficulties that occur when we learn the imbalanced data. Many real-world applications have to predict the solutions for highly imbalanced datasets with the imbalanced target variable. In most of the cases, the target variable assigns or having the least occurrences of the target values due to the sort of imbalances associated with things or events strongly applicable for the users who avail the solutions (for example, results of stock changes, fraud finding, network security, etc.). The expansion of the availability of data due to the rise of big data from the network systems such as security, internet transactions, finance manipulations, surveillance of CCTV or other devices makes the chance to the critical study of insufficient knowledge from the imbalance data when supporting the decision making processes. The data imbalance occurrence is a challenge to the research field. In recent trends, there is more data level and an algorithm level method is being upgraded constantly and leads to develop a new hybrid framework to solve this problem in classification. Classifying the imbalanced data is a challenging task in the field of big data analytics. This study mainly concentrates on the problem existing in most cases of real-world applications as an imbalance occurs in the data. This difficulty present due to the data distribution with skewed nature. We have analyses the data imbalance and find the solution. This paper concentrates mainly on finding a better solution to this nature of the problem to be solved with the proposed framework using a hybrid ensemble classifier based on the Binary Cross-Entropy method as loss function along with the Gradient Boost Algorithm.
3
Yogi, Abhishek, and Ratul Dey. "CLASS IMBALANCE PROBLEM IN DATA SCIENCE: REVIEW." International Research Journal of Computer Science 9, no. 4 (April 30, 2022): 56–60. http://dx.doi.org/10.26562/irjcs.2021.v0904.002.
Full textAbstract:
In last few years there are many changes and evolution has been done on the classification of data. Application area of technology increases then the size of data also increases. Classification of data becomes difficult because of imbalance nature and unbounded size of data. Class imbalance problem becomes the greatest issue in the data mining. Imbalance problem occurs where one of the two classes having more sample than the other classes. The most of the algorithms are focusing on the classification of major sample while ignoring minority sample. The minority samples are those samples that rarely occur but are very important. There are different methods available for the classification of imbalance data set which is divided into three categories i.e. the algorithmic approach, feature selection approach and the data processing approach. These approaches have their own advantages and disadvantages. In this paper a systematic study of each process is defined which gives the right direction for research in class imbalance problem.
4
Rendón, Eréndira, Roberto Alejo, Carlos Castorena, Frank J. Isidro-Ortega, and Everardo E. Granda-Gutiérrez. "Data Sampling Methods to Deal With the Big Data Multi-Class Imbalance Problem." Applied Sciences 10, no. 4 (February 14, 2020): 1276. http://dx.doi.org/10.3390/app10041276.
Full textAbstract:
The class imbalance problem has been a hot topic in the machine learning community in recent years. Nowadays, in the time of big data and deep learning, this problem remains in force. Much work has been performed to deal to the class imbalance problem, the random sampling methods (over and under sampling) being the most widely employed approaches. Moreover, sophisticated sampling methods have been developed, including the Synthetic Minority Over-sampling Technique (SMOTE), and also they have been combined with cleaning techniques such as Editing Nearest Neighbor or Tomek’s Links (SMOTE+ENN and SMOTE+TL, respectively). In the big data context, it is noticeable that the class imbalance problem has been addressed by adaptation of traditional techniques, relatively ignoring intelligent approaches. Thus, the capabilities and possibilities of heuristic sampling methods on deep learning neural networks in big data domain are analyzed in this work, and the cleaning strategies are particularly analyzed. This study is developed on big data, multi-class imbalanced datasets obtained from hyper-spectral remote sensing images. The effectiveness of a hybrid approach on these datasets is analyzed, in which the dataset is cleaned by SMOTE followed by the training of an Artificial Neural Network (ANN) with those data, while the neural network output noise is processed with ENN to eliminate output noise; after that, the ANN is trained again with the resultant dataset. Obtained results suggest that best classification outcome is achieved when the cleaning strategies are applied on an ANN output instead of input feature space only. Consequently, the need to consider the classifier’s nature when the classical class imbalance approaches are adapted in deep learning and big data scenarios is clear.
5
SUN, YANMIN, ANDREW K. C. WONG, and MOHAMED S. KAMEL. "CLASSIFICATION OF IMBALANCED DATA: A REVIEW." International Journal of Pattern Recognition and Artificial Intelligence 23, no. 04 (June 2009): 687–719. http://dx.doi.org/10.1142/s0218001409007326.
Full textAbstract:
Classification of data with imbalanced class distribution has encountered a significant drawback of the performance attainable by most standard classifier learning algorithms which assume a relatively balanced class distribution and equal misclassification costs. This paper provides a review of the classification of imbalanced data regarding: the application domains; the nature of the problem; the learning difficulties with standard classifier learning algorithms; the learning objectives and evaluation measures; the reported research solutions; and the class imbalance problem in the presence of multiple classes.
6
Liu, Tian Yu. "Research on Feature Selection for Imbalanced Problem from Fault Diagnosis on Gear." Advanced Materials Research 466-467 (February 2012): 886–90. http://dx.doi.org/10.4028/www.scientific.net/amr.466-467.886.
Full textAbstract:
Defect is one of the important factors resulting in gear fault, so it is significant to study the technology of defect diagnosis for gear. Class imbalance problem is encountered in the fault diagnosis, which causes seriously negative effect on the performance of classifiers that assume a balanced distribution of classes. Though it is critical, few previous works paid attention to this class imbalance problem in the fault diagnosis of gear. In imbalanced problems, some features are redundant and even irrelevant. These features will hurt the generalization performance of learning machines. Here we propose PREE (Prediction Risk based feature selectionfor EasyEnsemble) to solve the class imbalanced problem in the fault diagnosis of gear. Experimental results on UCI data sets and gear data set show that PREE improves the classification performance and prediction ability on the imbalanced dataset.
7
Khoshgoftaar, Taghi M., Naeem Seliya, and Dennis J. Drown. "Evolutionary data analysis for the class imbalance problem." Intelligent Data Analysis 14, no. 1 (January 22, 2010): 69–88. http://dx.doi.org/10.3233/ida-2010-0409.
Full text
8
., Hartono, Opim Salim Sitompul, Erna Budhiarti Nababan, Tulus ., Dahlan Abdullah, and Ansari Saleh Ahmar. "A New Diversity Technique for Imbalance Learning Ensembles." International Journal of Engineering & Technology 7, no. 2.14 (April 8, 2018): 478. http://dx.doi.org/10.14419/ijet.v7i2.11251.
Full textAbstract:
Data mining and machine learning techniques designed to solve classification problems require balanced class distribution. However, in reality sometimes the classification of datasets indicates the existence of a class represented by a large number of instances whereas there are classes with far fewer instances. This problem is known as the class imbalance problem. Classifier Ensembles is a method often used in overcoming class imbalance problems. Data Diversity is one of the cornerstones of ensembles. An ideal ensemble system should have accurrate individual classifiers and if there is an error it is expected to occur on different objects or instances. This research will present the results of overview and experimental study using Hybrid Approach Redefinition (HAR) Method in handling class imbalance and at the same time expected to get better data diversity. This research will be conducted using 6 datasets with different imbalanced ratios and will be compared with SMOTEBoost which is one of the Re-Weighting method which is often used in handling class imbalance. This study shows that the data diversity is related to performance in the imbalance learning ensembles and the proposed methods can obtain better data diversity.
9
Naboureh, Amin, Ainong Li, Jinhu Bian, Guangbin Lei, and Meisam Amani. "A Hybrid Data Balancing Method for Classification of Imbalanced Training Data within Google Earth Engine: Case Studies from Mountainous Regions." Remote Sensing 12, no. 20 (October 11, 2020): 3301. http://dx.doi.org/10.3390/rs12203301.
Full textAbstract:
Distribution of Land Cover (LC) classes is mostly imbalanced with some majority LC classes dominating against minority classes in mountainous areas. Although standard Machine Learning (ML) classifiers can achieve high accuracies for majority classes, they largely fail to provide reasonable accuracies for minority classes. This is mainly due to the class imbalance problem. In this study, a hybrid data balancing method, called the Partial Random Over-Sampling and Random Under-Sampling (PROSRUS), was proposed to resolve the class imbalance issue. Unlike most data balancing techniques which seek to fully balance datasets, PROSRUS uses a partial balancing approach with hundreds of fractions for majority and minority classes to balance datasets. For this, time-series of Landsat-8 and SRTM topographic data along with various spectral indices and topographic data were used over three mountainous sites within the Google Earth Engine (GEE) cloud platform. It was observed that PROSRUS had better performance than several other balancing methods and increased the accuracy of minority classes without a reduction in overall classification accuracy. Furthermore, adopting complementary information, particularly topographic data, considerably increased the accuracy of minority classes in mountainous areas. Finally, the obtained results from PROSRUS indicated that every imbalanced dataset requires a specific fraction(s) for addressing the class imbalance problem, because different datasets contain various characteristics.
10
Liu, Zhenyan, Yifei Zeng, Pengfei Zhang, Jingfeng Xue, Ji Zhang, and Jiangtao Liu. "An Imbalanced Malicious Domains Detection Method Based on Passive DNS Traffic Analysis." Security and Communication Networks 2018 (June 20, 2018): 1–7. http://dx.doi.org/10.1155/2018/6510381.
Full textAbstract:
Although existing malicious domains detection techniques have shown great success in many real-world applications, the problem of learning from imbalanced data is rarely concerned with this day. But the actual DNS traffic is inherently imbalanced; thus how to build malicious domains detection model oriented to imbalanced data is a very important issue worthy of study. This paper proposes a novel imbalanced malicious domains detection method based on passive DNS traffic analysis, which can effectively deal with not only the between-class imbalance problem but also the within-class imbalance problem. The experiments show that this proposed method has favorable performance compared to the existing algorithms.
Dissertations / Theses on the topic "Data imbalance problem"
1
Gao, Jie. "Data Augmentation in Solving Data Imbalance Problems." Thesis, KTH, Skolan för elektroteknik och datavetenskap (EECS), 2020. http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-289208.
Full textAbstract:
This project mainly focuses on the various methods of solving data imbalance problems in the Natural Language Processing (NLP) field. Unbalanced text data is a common problem in many tasks especially the classification task, which leads to the model not being able to predict the minority class well. Sometimes, even we change to some more excellent and complicated model could not improve the performance, while some simple data strategies that focus on solving data imbalanced problems such as over-sampling or down-sampling produce positive effects on the result. The common data strategies include some re-sampling methods that duplicate new data from the original data or remove some original data to have the balance. Except for that, some other methods such as word replacement, word swap, and word deletion are used in previous work as well. At the same time, some deep learning models like BERT, GPT and fastText model, which have a strong ability for a general understanding of natural language, so we choose some of them to solve the data imbalance problem. However, there is no systematic comparison in practicing these methods. For example, over-sampling and down-sampling are fast and easy to use in previous small scales of datasets. With the increase of the dataset, the newly generated data by some deep network models is more compatible with the original data. Therefore, our work focus on how is the performance of various data augmentation techniques when they are used to solve data imbalance problems, given the dataset and task? After the experiment, Both qualitative and quantitative experimental results demonstrate that different methods have their advantages for various datasets. In general, data augmentation could improve the performance of classification models. For specific, BERT especially our fine-tuned BERT has an excellent ability in most using scenarios(different scales and types of the dataset). Still, other techniques such as Back-translation has a better performance in long text data, even it costs more time and has a complicated model. In conclusion, suitable choices for data augmentation methods could help to solve data imbalance problems.Detta projekt fokuserar huvudsakligen på de olika metoderna för att lösa dataobalansproblem i fältet Natural Language Processing (NLP). Obalanserad textdata är ett vanligt problem i många uppgifter, särskilt klassificeringsuppgiften, vilket leder till att modellen inte kan förutsäga minoriteten Ibland kan vi till och med byta till en mer utmärkt och komplicerad modell inte förbättra prestandan, medan några enkla datastrategier som fokuserar på att lösa data obalanserade problem som överprov eller nedprovning ger positiva effekter på resultatet. vanliga datastrategier inkluderar några omprovningsmetoder som duplicerar nya data från originaldata eller tar bort originaldata för att få balans. Förutom det används vissa andra metoder som ordbyte, ordbyte och radering av ord i tidigare arbete Samtidigt har vissa djupinlärningsmodeller som BERT, GPT och fastText-modellen, som har en stark förmåga till en allmän förståelse av naturliga språk, så vi väljer några av dem för att lösa problemet med obalans i data. Det finns dock ingen systematisk jämförelse när man praktiserar dessa metoder. Exempelvis är överprovtagning och nedprovtagning snabba och enkla att använda i tidigare små skalor av datamängder. Med ökningen av datauppsättningen är de nya genererade data från vissa djupa nätverksmodeller mer kompatibla med originaldata. Därför fokuserar vårt arbete på hur prestandan för olika dataförstärkningstekniker används när de används för att lösa dataobalansproblem, givet datamängden och uppgiften? Efter experimentet visar både kvalitativa och kvantitativa experimentella resultat att olika metoder har sina fördelar för olika datamängder. I allmänhet kan dataförstärkning förbättra prestandan hos klassificeringsmodeller. För specifika, BERT speciellt vår finjusterade BERT har en utmärkt förmåga i de flesta med hjälp av scenarier (olika skalor och typer av datamängden). Ändå har andra tekniker som Back-translation bättre prestanda i lång textdata, till och med det kostar mer tid och har en komplicerad modell. Sammanfattningsvis lämpliga val för metoder för dataökning kan hjälpa till att lösa problem med obalans i data.
2
Barella, Victor Hugo. "Técnicas para o problema de dados desbalanceados em classificação hierárquica." Universidade de São Paulo, 2015. http://www.teses.usp.br/teses/disponiveis/55/55134/tde-06012016-145045/.
Full textAbstract:
Os recentes avanços da ciência e tecnologia viabilizaram o crescimento de dados em quantidade e disponibilidade. Junto com essa explosão de informações geradas, surge a necessidade de analisar dados para descobrir conhecimento novo e útil. Desse modo, áreas que visam extrair conhecimento e informações úteis de grandes conjuntos de dados se tornaram grandes oportunidades para o avanço de pesquisas, tal como o Aprendizado de Máquina (AM) e a Mineração de Dados (MD). Porém, existem algumas limitações que podem prejudicar a acurácia de alguns algoritmos tradicionais dessas áreas, por exemplo o desbalanceamento das amostras das classes de um conjunto de dados. Para mitigar tal problema, algumas alternativas têm sido alvos de pesquisas nos últimos anos, tal como o desenvolvimento de técnicas para o balanceamento artificial de dados, a modificação dos algoritmos e propostas de abordagens para dados desbalanceados. Uma área pouco explorada sob a visão do desbalanceamento de dados são os problemas de classificação hierárquica, em que as classes são organizadas em hierarquias, normalmente na forma de árvore ou DAG (Direct Acyclic Graph). O objetivo deste trabalho foi investigar as limitações e maneiras de minimizar os efeitos de dados desbalanceados em problemas de classificação hierárquica. Os experimentos realizados mostram que é necessário levar em consideração as características das classes hierárquicas para a aplicação (ou não) de técnicas para tratar problemas dados desbalanceados em classificação hierárquica.Recent advances in science and technology have made possible the data growth in quantity and availability. Along with this explosion of generated information, there is a need to analyze data to discover new and useful knowledge. Thus, areas for extracting knowledge and useful information in large datasets have become great opportunities for the advancement of research, such as Machine Learning (ML) and Data Mining (DM). However, there are some limitations that may reduce the accuracy of some traditional algorithms of these areas, for example the imbalance of classes samples in a dataset. To mitigate this drawback, some solutions have been the target of research in recent years, such as the development of techniques for artificial balancing data, algorithm modification and new approaches for imbalanced data. An area little explored in the data imbalance vision are the problems of hierarchical classification, in which the classes are organized into hierarchies, commonly in the form of tree or DAG (Direct Acyclic Graph). The goal of this work aims at investigating the limitations and approaches to minimize the effects of imbalanced data with hierarchical classification problems. The experimental results show the need to take into account the features of hierarchical classes when deciding the application of techniques for imbalanced data in hierarchical classification.
3
Gao, Ming. "A study on imbalanced data classification problems." Thesis, University of Reading, 2013. http://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.602707.
Full textAbstract:
This thesis focuses on the study of machine learning and pattern recognition algorithms for imbalanced data problems. The imbalanced problems are important as they are prevalent in life threatening/safety critical applications. They are known to be problematic to standard machine learning algorithms due to the imbalanced distribution between positive and negative classes. My original contribution to knowledge in this field is fourfold. A powerful and efficient algorithm for solving two-class imbalanced problems is proposed. The proposed method combines the synthetic minority over-sampling technique and the radial basis function classifier optimised by particle swarm optimization to enhance the classifier's performance for imbalanced learning. An over-sampling technique for imbalanced problems, probability density function estimation based over-sampling, is proposed. In contrast to existing over-sampling techniques that lack sufficient theoretical insights and justifications, the synthetic data samples are generated from the estimated probability density function from the positive data via the Parzen-window. A unified neurofuzzy modelling scheme is proposed. A novel initial rule construction method on the subspaces of the input features is formed. The supervised subspace orthogonal least square learning for model construction is applied. A logistic regression model is formed to present the classifiers output. Based on the formation of the unified neurofuzzy model, a new class of neurofuzzy construction algorithms is proposed with the aim of maximizing generalization capability specifically for imbalanced data classification based on leave-one-out cross-validation.
4
Jeatrakul, Piyasak. "Enhancing classification performance over noise and imbalanced data problems." Thesis, Jeatrakul, Piyasak (2012) Enhancing classification performance over noise and imbalanced data problems. PhD thesis, Murdoch University, 2012. https://researchrepository.murdoch.edu.au/id/eprint/10044/.
Full textAbstract:
This research presents the development of techniques to handle two issues in data classification: noise and imbalanced data problems. Noise is a significant problem that can degrade the quality of training data in any learning algorithm. Learning algorithms trained by noisy instances generally increase misclassification when they perform classification. As a result, the classification performance tends to decrease. Meanwhile, the imbalanced data problem is another problem affecting the performance of learning algorithms. If some classes have a much larger number of instances than the others, the learning algorithms tend to be dominated by the features of the majority classes, and the features of the minority classes are difficult to recognise. As a result, the classification performance of the minority classes could be significantly lower than that of the majority classes. It is therefore important to implement techniques to better handle the negative effects of noise and imbalanced data problems.
Although there are several approaches attempting to handle noise and imbalanced data problems, shortcomings of the available approaches still exist. For the noise handling techniques, even though the noise tolerant approach does not require any data preprocessing, it can tolerate only a certain amount of noise. The classifier developed from noisy data tends to be less predictive if the training data contains a great number of noise instances. Furthermore, for the noise elimination approach, although it can be easily applied to various problem domains, it could degrade the quality of training data if it cannot distinguish between noise and rare cases (exceptions). Besides, for the imbalanced data problem, the available techniques used still present some limitations. For example, the algorithm-level approach can perform effectively only on specific problem domains or specific learning algorithms. The data-level approach can either eliminate necessary information from the training set or produce the over-fitting problem over the minority class. Moreover, when the imbalanced data problem becomes more complex, such as for the case of multi-class classification, it is difficult to apply the re-sampling techniques (the data-level approach), which perform effectively for imbalanced data problems in binary classification, to the multi-class classification. Due to the limitations above, these lead to the motivation of this research to propose and investigate techniques to handle noise and imbalanced data problems more effectively.
This thesis has developed three new techniques to overcome the identified problems. Firstly, a cleaning technique called the Complementary Neural Network (CMTNN) data cleaning technique has been developed in order to remove noise (misclassification data) from the training set. The results show that the new noise detection and removal technique can eliminate noise with confidence. Furthermore, the CMTNN cleaning technique can increase the classification accuracy across different learning algorithms, which are Artificial Neural Network (ANN), Support Vector Machine (SVM), k- Nearest Neighbor (k-NN), and Decision Tree (DT). It can provide higher classification performance than other cleaning methods such as Tomek links, the majority voting filtering, and the consensus voting filtering.
Secondly, the CMTNN re-sampling technique, which is a new under-sampling technique, has been developed to handle the imbalanced data problem in binary classification. The results show that the combined techniques of the CMTNN resampling technique and Synthetic Minority Over-sampling Technique (SMOTE) can perform effectively by improving the classification performance of the minority class instances in terms of Geometric Mean (G-Mean) and the area under the Receiver Operating Characteristic (ROC) curve. It generally provides higher performance than other re-sampling techniques such as Tomek links, Wilson’s Edited Nearest Neighbor Rule (ENN), SMOTE, the combined technique of SMOTE and ENN, and the combined technique of SMOTE and Tomek links.
For the third proposed technique, an algorithm named One-Against-All with Data Balancing (OAA-DB) has been developed in order to deal with the imbalanced data problem in multi-class classification. It can be asserted that this algorithm not only improves the performance for the minority class but it also maintains the overall accuracy, which is normally reduced by other techniques. The OAA-DB algorithm can increase the performance in terms of the classification accuracy and F-measure when compared to other multi-class classification approaches including One-Against-All (OAA), One-Against-One (OAO), All and One (A&O), and One Against Higher Order (OAHO) approaches. Furthermore, this algorithm has shown that the re-sampling technique is not only used effectively for the class imbalance problem in binary classification but it has been also applied successfully to the imbalanced data problem in multi-class classification.
5
Pan, Yi-Ying, and 潘怡瑩. "Clustering-based Data Preprocessing Approach for the Class Imbalance Problem." Thesis, 2018. http://ndltd.ncl.edu.tw/handle/94nys8.
Full textAbstract:
碩士國立中央大學
資訊管理學系
106
The class imbalance problem is an important issue in data mining. It occurs when the number of samples in one class is much larger than the other classes. Traditional classifiers tend to misclassify most samples of the minority class into the majority class for maximizing the overall accuracy. This phenomenon makes it hard to establish a good classification rule for the minority class. The class imbalance problem often occurs in many real world applications, such as fault diagnosis, medical diagnosis and face recognition. To deal with the class imbalance problem, a clustering-based data preprocessing approach is proposed, where two different clustering techniques including affinity propagation clustering and K-means clustering are used individually to divide the majority class into several subclasses resulting in multiclass data. This approach can effectively reduce the class imbalance ratio of the training dataset, shorten the class training time and improve classification performance. Our experiments based on forty-four small class imbalance datasets from KEEL and eight high-dimensional datasets from NASA to build five types of classification models, which are C4.5, MLP, Naïve Bayes, SVM and k-NN (k=5). In addition, we also employ the classifier ensemble algorithm. This research tries to compare AUC results between different clustering techniques, different classification models and the number of clusters of K-means clustering in order to find out the best configuration of the proposed approach and compare with other literature methods. Finally, the experimental results of the KEEL datasets show that k-NN (k=5) algorithm is the best choice regardless of whether affinity propagation or K-means (K=5); the experimental results of NASA datasets show that the performance of the proposed approach is superior to the literature methods for the high-dimensional datasets.
6
Komba, Lyee, and Lyee Komba. "Sampling Techniques for Class Imbalance Problem in Aviation Safety Incidents Data." Thesis, 2018. http://ndltd.ncl.edu.tw/handle/jg2y52.
Full textAbstract:
碩士國立臺北科技大學
電資國際專班
106
Like any other industries in the world, the aviation industry has a variety data acquired everyday through numerous data management systems. Structured and unstructured data are being collected through aircraft systems, maintenance systems, supply systems, ticketing and booking systems, and many other systems that are utilized in the daily operations of aviation business. Data mining can be used to analyze all these different types of data to generate meaningful information that can improve future performance, safety and profitability for aviation business and operations. This thesis presents details of data mining methods based on aviation incident data to predict incidents with fatal or a death consequence. Other literature have applied data mining techniques within the aviation industry include prediction of passenger travel, meteorological prediction, component failure prediction and other fatal incident prediction literature that aimed at finding the right features. This study uses the public dataset from the Federal Aviation Authority Accidents and Incidents Data System (FAA AIDS) website – data records from the year 2000 to year 2017. Our goal is to build a prediction model for fatal incidents and generate decision rules or factors contributing to incidents that have fatal results. In this way, the model to be built will be a predictive risk management system for aviation safety. The aviation industry generally operates at a safe state because of the transition from reactive safety and risk management to a proactive safety management approach; and now a predictive approach to safety management with the application of data mining techniques such as from this study and others. Over time, the number of systems has increased and the number of aviation accidents and serious incidents has decreased. Hence, a 0.6% of incidents with fatal consequences was attained from our analysis. During the data preprocessing stage, a problem of unbalanced dataset is encountered that invokes us to propose some techniques to solve the issue. Unbalanced datasets are datasets where least number of data is representing the minority classes than the majority class, especially when the analysis is aimed at the minority class. Not dealing with this issue correctly may result in poor performing models or misclassified data. With the increase of the travelling population in the aviation community, safety is paramount so coming up with a relatively precise model is important. In order to come up with a precise model/classifier, we need to preprocess and resample the data efficiently. This thesis also looks at combating the issue of unbalanced data to come up with a balanced data that can be used to train a classifier to design a precise model. We applied the following sampling technique in R Studio– oversampling, under-sampling, SMOTE and bootstrap samples to solve the imbalanced data. The resulting dataset from the unbalanced dataset resolution techniques are used to train different classifiers and the performance of the classifiers are measured and discussed in this thesis.
7
Yao, Guan-Ting, and 姚冠廷. "A Two-Stage Hybrid Data Preprocessing Approach for the Class Imbalance Problem." Thesis, 2017. http://ndltd.ncl.edu.tw/handle/dm48kk.
Full textAbstract:
碩士國立中央大學
資訊管理學系
105
The class imbalance problem is an important issue in data mining. The class skewed distribution occurs when the number of examples that represent one class is much lower than the ones of the other classes. The traditional classifiers tend to misclassify most samples in the minority class into the majority class because of maximizing the overall accuracy. This phenomenon limits the construction of effective classifiers for the precious minority class. This problem occurs in many real-world applications, such as fault diagnosis, medical diagnosis and face recognition. To deal with the class imbalance problem, I proposed a two-stage hybrid data preprocessing framework based on clustering and instance selection techniques. This approach filters out the noisy data in the majority class and can reduce the execution time for classifier training. More importantly, it can decrease the effect of class imbalance and perform very well in the classification task. Our experiments using 44 class imbalance datasets from KEEL to build four types of classification models, which are C4.5, k-NN, Naïve Bayes and MLP. In addition, the classifier ensemble algorithm is also employed. In addition, two kinds of clustering techniques and three kinds of instance selection algorithms are used in order to find out the best combination suited for the proposed method. The experimental results show that the proposed framework performs better than many well-known state-of-the-art approaches in terms of AUC. In particular, the proposed framework combined with bagging based MLP ensemble classifiers perform the best, which provide 92% of AUC.
8
吳思翰. "Combine Particle Swarm Optimization and Mahalonobis-Taguchi System for Solving Classification Problem in Imbalance Data." Thesis, 2010. http://ndltd.ncl.edu.tw/handle/06887158161687794935.
Full text
9
Chang, Yu-shan, and 張毓珊. "Developing Data Mining Models for Class Imbalance Problems." Thesis, 2009. http://ndltd.ncl.edu.tw/handle/57781951199735409394.
Full textAbstract:
碩士朝陽科技大學
資訊管理系碩士班
98
In classification problems, the class imbalance problem would cause a bias on the training of classifiers and result in a low predictive accuracy over the minority class examples. This problem is caused by imbalanced data in which almost all examples belong to one class and far fewer instances belong to others. Compared with the majority examples, the minority examples are usually more interesting class, such as rare diseases in medical diagnosis data, failures in inspection data, frauds in credit screening data, and so on. When inducing knowledge from an imbalanced data set, traditional data mining algorithms will seek high classification accuracy for the majority class, but an unacceptable error rate for the minority class. Therefore, they are not suitable for handling the class imbalanced data. In order to tackle the class imbalance problem, this study aims to (1) find a robust classifier from different candidates including Decision Tree (DT), Logistic Regression (LR), Mahalanobis Distance (MD), and Support Vector Machines (SVM); (2) propose two novel methods called MD-SVM (a new two-phase learning scheme) and SWAI (SOM Weights As Input). Experimental results indicated our proposed MD-SVM and SWAI has better performance in identifying the minority class examples compared with traditional techniques such as under-sampling, cost adjusting, and cluster based sampling.
10
Liu, Yi-Hsun, and 劉奕勛. "Deep Discriminative Features Learning and Sampling for Imbalanced Data Problem." Thesis, 2018. http://ndltd.ncl.edu.tw/handle/3cc7k8.
Full textAbstract:
碩士國立交通大學
資訊科學與工程研究所
106
The imbalanced data problem occurs in many application domains and is considered to be a challenging problem in machine learning and data mining. Oversampling may lead to overfitting, while undersampling may discard representative data samples. Additionally, most resampling methods for synthetic data focus on minority class without considering the data distribution of major classes. This paper presents an algorithm that combines feature embedding with the loss functions from discriminative feature learning in deep learning to generate synthetic data samples. In contrast to previous works, the proposed method considers both majority classes and minority classes to learn feature embeddings and utilizes appropriate loss functions to make feature embedding as discriminative as possible. The proposed method is a comprehensive framework and different feature extractors can be utilized for different domains. We conduct experiments utilizing eight numerical datasets and one image dataset based on multiclass classification tasks. The experimental results indicate that the proposed method provides accurate and stable results. Additionally, we thoroughly investigate the proposed method and utilize a visualization technique to determine why the proposed method can generate good data samples.
Books on the topic "Data imbalance problem"
1
Zabelina, Ol'ga, Irina Omel'chenko, Anna Mayorova, and Ekaterina Safonova. Human resource Development in the Digital Age: Strategic Challenges, Challenges, and Opportunities. ru: INFRA-M Academic Publishing LLC., 2021. http://dx.doi.org/10.12737/1243772.
Full textAbstract:
The monograph, based on the identification of trends and problems of changes in the demand and supply of skills, as well as the study of modern mechanisms of their formation and actualization, substantiates the priority areas of human resources development in the Russian Federation that meet the strategic challenges of the period of digital transformation of the labor sphere.
The authors identify and systematize current and future trends related to changes in the demand for professions and skills in the Russian and global labor markets. The directions of transformation of the demand for skills and professions in the conditions of digitalization of the economy, skills and professions of the future are determined. Quantitative and qualitative imbalances and trends in labor supply and demand in the Russian labor market are identified (based on statistical analysis of data from 2009-2019). The features and problems of supply and demand of professions/skills in the segments of the Russian labor market covered by Internet recruitment are identified (based on data from resume parsing and vacancies of Internet recruitment portals in 2018 and 2020). Methodological approaches to identifying widely-and poorly-demanded skills are proposed and tested during the competence analysis of labor supply and demand using Big Data technologies.the competence profile of the vacancies of the professional core and extra - skills. An innovative author's approach to assessing the potential of skills capitalization — a possible increase in the salary of an applicant due to the expansion of the set of skills that he has-is proposed and tested. The current policy directions of formation and improvement of skills of the population in the Russian Federation are identified and systematized. The strategic challenges of the period of digital transformation of the labor sphere facing the Russian Federation and the priority areas of human resources development that meet these challenges are identified.
The conclusions and recommendations can be used in the work of the Ministry of Labor of Russia, Rostrud, the Ministry of Education and Science of Russia, the Ministry of Education of Russia, government authorities, employment services of the Russian regions, as well as organizations of the professional education system.
2
Bennett, Jeremy, and Kara Siegrist. Myocardial Ischemia. Edited by Matthew D. McEvoy and Cory M. Furse. Oxford University Press, 2017. http://dx.doi.org/10.1093/med/9780190226459.003.0005.
Full textAbstract:
Coronary artery disease is a prevalent and growing problem in the United States leading to significant morbidity and mortality including myocardial ischemia and infarction. Diagnosis and treatment of myocardial ischemia under general anesthesia can present unique challenges for the anesthesiologist including interpretation of diagnostic monitoring data and options for therapeutic interventions. There are many complex factors that determine myocardial oxygen supply and demand; when these become imbalanced, myocardial ischemia occurs that can progress to infarction. Maintaining a high-degree of suspicion for myocardial events in the perioperative period is paramount to good patient outcomes. In fact, perioperative myocardial injury within 30 days of surgery, if considered as a disease entity, would be the third leading cause of death in the United States. This chapters reviews the diagnosis and treatment of such events.
3
Kirchman, David L. The nitrogen cycle. Oxford University Press, 2018. http://dx.doi.org/10.1093/oso/9780198789406.003.0012.
Full textAbstract:
Nitrogen is required for the biosynthesis of many cellular components and can take on many oxidation states, ranging from −3 to +5. Consequently, nitrogen compounds can act as either electron donors (chemolithotrophy) or electron acceptors (anaerobic respiration). The nitrogen cycle starts with nitrogen fixation, the reduction of nitrogen gas to ammonium. Nitrogen fixation is carried out only by prokaryotes, mainly some cyanobacteria and heterotrophic bacteria. The ammonium resulting from nitrogen fixation is quickly used by many organisms for biosynthesis, being preferred over nitrate as a nitrogen source. It is also oxidized aerobically by chemolithoautotrophic bacteria and archaea during the first step of nitrification. The second step, nitrite oxidation, is carried out by other bacteria not involved in ammonia oxidation, resulting in the formation of nitrate. Some bacteria are capable of carrying out both steps (“comammox”). This nitrate can then be reduced to nitrogen gas or nitrous oxide during denitrification. It can be reduced to ammonium, a process called “dissimilatory nitrate reduction to ammonium.” Nitrogen gas is also released by anaerobic oxidation of ammonium (“anammox”) which is carried out by bacteria in the Planctomycetes phylum. The theoretical contribution of anammox to total nitrogen gas release is 29%, but the actual contribution varies greatly. Another gas in the nitrogen cycle, nitrous oxide, is a greenhouse gas produced by ammonia-oxidizing bacteria and archaea. The available data indicate that the global nitrogen cycle is in balance, with losses from nitrogen gas production equaling gains via nitrogen fixation. But excess nitrogen from fertilizers is contributing to local imbalances and several environmental problems in drinking waters, reservoirs, lakes, and coastal oceans.
Book chapters on the topic "Data imbalance problem"
1
Ling, Charles X., and Victor S. Sheng. "Class Imbalance Problem." In Encyclopedia of Machine Learning and Data Mining, 204–5. Boston, MA: Springer US, 2017. http://dx.doi.org/10.1007/978-1-4899-7687-1_110.
Full text
2
Gosain, Anjana, Arushi Gupta, and Deepika Singh. "Hybrid Data-Level Techniques for Class Imbalance Problem." In Advances in Intelligent Systems and Computing, 1131–41. Singapore: Springer Singapore, 2020. http://dx.doi.org/10.1007/978-981-15-5113-0_95.
Full text
3
Kozal, Jȩdrzej, and Paweł Ksieniewicz. "Imbalance Reduction Techniques Applied to ECG Classification Problem." In Intelligent Data Engineering and Automated Learning – IDEAL 2019, 323–31. Cham: Springer International Publishing, 2019. http://dx.doi.org/10.1007/978-3-030-33617-2_33.
Full text
4
Hoens, T. Ryan, Qi Qian, Nitesh V. Chawla, and Zhi-Hua Zhou. "Building Decision Trees for the Multi-class Imbalance Problem." In Advances in Knowledge Discovery and Data Mining, 122–34. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012. http://dx.doi.org/10.1007/978-3-642-30217-6_11.
Full text
5
Himaja, D., T. Maruthi Padmaja, and P. Radha Krishna. "Oversample Based Large Scale Support Vector Machine for Online Class Imbalance Problem." In Big Data Analytics, 348–62. Cham: Springer International Publishing, 2018. http://dx.doi.org/10.1007/978-3-030-04780-1_24.
Full text
6
Hoens, T. Ryan, and Nitesh V. Chawla. "Generating Diverse Ensembles to Counter the Problem of Class Imbalance." In Advances in Knowledge Discovery and Data Mining, 488–99. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010. http://dx.doi.org/10.1007/978-3-642-13672-6_46.
Full text
7
Sasirekha, R., B. Kanisha, and S. Kaliraj. "Study on Class Imbalance Problem with Modified KNN for Classification." In Intelligent Data Communication Technologies and Internet of Things, 207–17. Singapore: Springer Singapore, 2022. http://dx.doi.org/10.1007/978-981-16-7610-9_15.
Full text
8
Malhotra, Ruchika, and Kusum Lata. "Tackling the Imbalanced Data in Software Maintainability Prediction Using Ensembles for Class Imbalance Problem." In Advances in Interdisciplinary Research in Engineering and Business Management, 391–99. Singapore: Springer Singapore, 2021. http://dx.doi.org/10.1007/978-981-16-0037-1_31.
Full text
9
Zangari, Murilo, Wesley Romão, and Ademir Aparecido Constantino. "Extensions of Ant-Miner Algorithm to Deal with Class Imbalance Problem." In Intelligent Data Engineering and Automated Learning - IDEAL 2012, 9–18. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012. http://dx.doi.org/10.1007/978-3-642-32639-4_2.
Full text
10
Al_Janabi, Samaher, and Fatma Razaq. "A Novel Tool DSMOTE to Handel Imbalance Customer Churn Problem in Telecommunication Industry." In Big Data and Networks Technologies, 36–50. Cham: Springer International Publishing, 2019. http://dx.doi.org/10.1007/978-3-030-23672-4_4.
Full textConference papers on the topic "Data imbalance problem"
1
Li, Yanling, Guoshe Sun, and Yehang Zhu. "Data Imbalance Problem in Text Classification." In 2010 Third International Symposium on Information Processing (ISIP). IEEE, 2010. http://dx.doi.org/10.1109/isip.2010.47.
Full text
2
Sarmanova, Akkenzhe, and S. Albayrak. "Alleviating class imbalance problem in data mining." In 2013 21st Signal Processing and Communications Applications Conference (SIU). IEEE, 2013. http://dx.doi.org/10.1109/siu.2013.6531574.
Full text
3
Johnson, Reid A., Nitesh V. Chawla, and Jessica J. Hellmann. "Species distribution modeling and prediction: A class imbalance problem." In 2012 Conference on Intelligent Data Understanding (CIDU). IEEE, 2012. http://dx.doi.org/10.1109/cidu.2012.6382186.
Full text
4
Wang, Jing, and Min-Ling Zhang. "Towards Mitigating the Class-Imbalance Problem for Partial Label Learning." In KDD '18: The 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY, USA: ACM, 2018. http://dx.doi.org/10.1145/3219819.3220008.
Full text
5
An, Chunsheng, Jingtong Sun, Yifeng Wang, and Qingjie Wei. "A K-means Improved CTGAN Oversampling Method for Data Imbalance Problem." In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS). IEEE, 2021. http://dx.doi.org/10.1109/qrs54544.2021.00097.
Full text
6
Su, Guangxin, Weitong Chen, and Miao Xu. "Positive-Unlabeled Learning from Imbalanced Data." In Thirtieth International Joint Conference on Artificial Intelligence {IJCAI-21}. California: International Joint Conferences on Artificial Intelligence Organization, 2021. http://dx.doi.org/10.24963/ijcai.2021/412.
Full textAbstract:
Positive-unlabeled (PU) learning deals with the binary classification problem when only positive (P) and unlabeled (U) data are available, without negative (N) data. Existing PU methods perform well on the balanced dataset. However, in real applications such as financial fraud detection or medical diagnosis, data are always imbalanced. It remains unclear whether existing PU methods can perform well on imbalanced data. In this paper, we explore this problem and propose a general learning objective for PU learning targeting specially at imbalanced data. By this general learning objective, state-of-the-art PU methods based on optimizing a consistent risk can be adapted to conquer the imbalance. We theoretically show that in expectation, optimizing our learning objective is equivalent to learning a classifier on the oversampled balanced data with both P and N data available, and further provide an estimation error bound. Finally, experimental results validate the effectiveness of our proposal compared to state-of-the-art PU methods.
7
Rashu, Raisul Islam, Naheena Haq, and Rashedur M. Rahman. "Data mining approaches to predict final grade by overcoming class imbalance problem." In 2014 17th International Conference on Computer and Information Technology (ICCIT). IEEE, 2014. http://dx.doi.org/10.1109/iccitechn.2014.7073095.
Full text
8
Mwangi, Peter Irungu, Lawrence Nderu, Leah Mutanu, and Dorcas Gicuku Mwigereri. "Hybrid Ensemble Model for Handling Class Imbalance Problem in Big Data Analytics." In 2022 International Conference on Electrical, Computer and Energy Technologies (ICECET). IEEE, 2022. http://dx.doi.org/10.1109/icecet55527.2022.9872764.
Full text
9
Baro, Pranita, and Malaya Dutta Borah. "A Hybrid Resampling Approach to Handle Class Imbalance Problem and Missing Data." In 2022 IEEE 9th Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON). IEEE, 2022. http://dx.doi.org/10.1109/upcon56432.2022.9986452.
Full text
10
Zhang, Xiaowan, and Bao-Gang Hu. "Learning in the Class Imbalance Problem When Costs are Unknown for Errors and Rejects." In 2012 IEEE 12th International Conference on Data Mining Workshops. IEEE, 2012. http://dx.doi.org/10.1109/icdmw.2012.167.
Full textReports on the topic "Data imbalance problem"
1
Lurie, Susan, John Labavitch, Ruth Ben-Arie, and Ken Shackel. Woolliness in Peaches and Nectarines. United States Department of Agriculture, 1995. http://dx.doi.org/10.32747/1995.7570557.bard.
Full textAbstract:
The overall goal of the research was to understand the processes involved in the development of woolliness in peaches and nectarines. Four specific hypotheses were proposed and in the course of the research evidence was gathered t support two of them and to not support two others. The hypotheses and a summary of the evidence are outlined below. 1. That woolliness arises from an imbalance between the activities of the cell wall pectin degrading enzymes. Using 'Flavortop' nectarines and 'Hermoza' peaches as model systems, storage regimes were manipulated to induce or prevent woolliness. The expression (mRNA abundance), protein content (Western blotting), and activity of polygalacturonase (PG) and pectin esterase (PE) were followed. Expression of the enzymes was not different, but activity and the ratio between PG and PE activities were quite different in fruits developing woolliness or ripening normally. This was also examined by looking at the substrate, the pectin moiety of the cell wall, and i woolly fruit there were more high molecular weight pectins with regions of non-methylated galacturonic acid residues. Taking an in vitro approach it was found a) that PE activity was stable at 0oC while PG activity decreased; b) incubating the calcium pectate fraction of the cell wall with PE extracted from peaches caused the polymers to form a gel characteristic of the visual woolly symptoms in peaches. 2. That continued cell wall synthesis occurs during storage and contributes to structural changes i cell walls and improper dissolution and softening after storage. We tried to adapt our technique of adding 13C-glucose to fruit discs, which was used successfully to follow cell wall synthesis during tomato ripening. However, the difference in sugar content between the two fruits (4% in tomato and 12% in peach) meant that the 13C-glucose was much more diluted within the general metabolite pool. We were unable to see any cell wall synthesis which meant that either the dilution factor was too great, or that synthesis was not occurring. 3. That controlled atmosphere (CA) prevents woolliness by lowering all enzyme activities. CA was found to greatly reduce mRNA abundance of the cell wall enzymes compared to regular air storage. However, their synthesis and activity recovered during ripening after CA storage and did not after regular air storage. Therefore, CA prevented the inhibition of enzyme activation found in regular air storage. 4. That changes in cell wall turgor and membrane function are important events in the development of woolliness. Using a micro pressure probe, turgor was measured in cells of individual 'O'Henry' and 'CalRed' peaches which were woolly or healthy. The relationship between firmness and turgor was the same in both fruit conditions. These data indicate that the development and expression of woolliness are not associated with differences in membrane function, at least with regard to the factors that determine cell turgor pressure. In addition, during the period of the grant additional areas were explored. Encoglucanase, and enzyme metabolizing hemicellulose, was found to be highly expressed air stored, but not in unstored or CA stored fruit. Activity gels showed higher activity in air stored fruit as well. This is the first indication that other components of the cell wall may be involved in woolliness. The role of ethylene in woolliness development was also investigated at it was found a) that woolly fruits had decreased ability to produce ethylene, b) storing fruits in the presence of ethylene delayed the appearance of woolliness. This latter finding has implication for an inexpensive strategy for storing peaches and nectarines.