Journal articles: 'MapReduce programming model'

1

Zhang, Guigang, Chao Li, Yong Zhang, and Chunxiao Xing. "A Semantic++ MapReduce Parallel Programming Model." International Journal of Semantic Computing 08, no. 03 (September 2014): 279–99. http://dx.doi.org/10.1142/s1793351x14400091.

Full text

Abstract:

Big data is playing a more and more important role in every area such as medical health, internet finance, culture and education etc. How to process these big data efficiently is a huge challenge. MapReduce is a good parallel programming language to process big data. However, it has lots of shortcomings. For example, it cannot process complex computing. It cannot suit real-time computing. In order to overcome these shortcomings of MapReduce and its variants, in this paper, we propose a Semantic++ MapReduce parallel programming model. This study includes the following parts. (1) Semantic++ MapReduce parallel programming model. It includes physical framework of semantic++ MapReduce parallel programming model and logic framework of semantic++ MapReduce parallel programming model; (2) Semantic++ extraction and management method for big data; (3) Semantic++ MapReduce parallel programming computing framework. It includes semantic++ map, semantic++ reduce and semantic++ shuffle; (4) Semantic++ MapReduce for multi-data centers. It includes basic framework of semantic++ MapReduce for multi-data centers and semantic++ MapReduce application framework for multi-data centers; (5) A Case Study of semantic++ MapReduce across multi-data centers.

APA, Harvard, Vancouver, ISO, and other styles

2

Lämmel, Ralf. "Google’s MapReduce programming model — Revisited." Science of Computer Programming 70, no. 1 (January 2008): 1–30. http://dx.doi.org/10.1016/j.scico.2007.07.001.

Full text

APA, Harvard, Vancouver, ISO, and other styles

3

Retnowo, Murti. "Syncronize Data Using MapReduceModel Programming." International Journal of Engineering Technology and Natural Sciences 3, no. 2 (December 31, 2021): 82–88. http://dx.doi.org/10.46923/ijets.v3i2.140.

Full text

Abstract:

Research in the processing of the data shows that the larger data increasingly requires a longer time. Processing huge amounts of data on a single computer has limitations that can be overcome by parallel processing. This study utilized the MapReduce programming model data synchronization by duplicating the data from database client to database server. MapReduce is a programming model that was developed to speed up the processing of large data. MapReduce model application on the training process performed on data sharing that is adapted to number of sub-process (thread) and data entry to database server and displays data from data synchronization. The experiments were performed using data of 1,000, 10,000, 100,000 and 1,000,000 of data, and use the thread as much as 1, 5, 10, 15, 20 and 25 threads. The results showed that the use of MapReduce programming model can result in a faster time, but time to create many thread that many require a longer time. The results of the use of MapReduce programming model can provide time efficiency in synchronizing data both on a single database or a distributed database.

APA, Harvard, Vancouver, ISO, and other styles

4

Garg, Uttama. "Data Analytic Models That Redress the Limitations of MapReduce." International Journal of Web-Based Learning and Teaching Technologies 16, no. 6 (November 2021): 1–15. http://dx.doi.org/10.4018/ijwltt.20211101.oa7.

Full text

Abstract:

The amount of data in today’s world is increasing exponentially. Effectively analyzing Big Data is a very complex task. The MapReduce programming model created by Google in 2004 revolutionized the big-data comput-ing market. Nowadays the model is being used by many for scientific and research analysis as well as for commercial purposes. The MapReduce model however is quite a low-level progamming model and has many limitations. Active research is being undertaken to make models that overcome/remove these limitations. In this paper we have studied some popular data analytic models that redress some of the limitations of MapReduce; namely ASTERIX and Pregel (Giraph) We discuss these models briefly and through the discussion highlight how these models are able to overcome MapReduce’s limitations.

APA, Harvard, Vancouver, ISO, and other styles

5

Gao, Tilei, Ming Yang, Rong Jiang, Yu Li, and Yao Yao. "Research on Computing Efficiency of MapReduce in Big Data Environment." ITM Web of Conferences 26 (2019): 03002. http://dx.doi.org/10.1051/itmconf/20192603002.

Full text

Abstract:

The emergence of big data has brought a great impact on traditional computing mode, the distributed computing framework represented by MapReduce has become an important solution to this problem. Based on the big data, this paper deeply studies the principle and framework of MapReduce programming. On the basis of mastering the principle and framework of MapReduce programming, the time consumption of distributed computing framework MapReduce and traditional computing model is compared with concrete programming experiments. The experiment shows that MapReduce has great advantages in large data volume.

APA, Harvard, Vancouver, ISO, and other styles

6

Siddesh, G. M., Kavya Suresh, K. Y. Madhuri, Madhushree Nijagal, B. R. Rakshitha, and K. G. Srinivasa. "Optimizing Crawler4j using MapReduce Programming Model." Journal of The Institution of Engineers (India): Series B 98, no. 3 (August 12, 2016): 329–36. http://dx.doi.org/10.1007/s40031-016-0267-z.

Full text

APA, Harvard, Vancouver, ISO, and other styles

7

Zhang, Weidong, Boxin He, Yifeng Chen, and Qifei Zhang. "GMR: graph-compatible MapReduce programming model." Multimedia Tools and Applications 78, no. 1 (August 23, 2017): 457–75. http://dx.doi.org/10.1007/s11042-017-5102-2.

Full text

APA, Harvard, Vancouver, ISO, and other styles

8

Durairaj, M., and T. S. Poornappriya. "Importance of MapReduce for Big Data Applications: A Survey." Asian Journal of Computer Science and Technology 7, no. 1 (May 5, 2018): 112–18. http://dx.doi.org/10.51983/ajcst-2018.7.1.1817.

Full text

Abstract:

Significant regard for MapReduce framework has been trapped by a wide range of areas. It is presently a practical model for data-focused applications because of its basic interface of programming, high elasticity, and capacity to withstand the subjection to defects. Additionally, it is fit for preparing a high extent of data in Distributed Computing environments (DCE). MapReduce, on various events, has turned out to be material to a wide scope of areas. MapReduce is a parallel programming model and a related usage presented by Google. In the programming model, a client determines the calculation by two capacities, Map and Reduce. The basic MapReduce library consequently parallelizes the calculation and handles muddled issues like data dispersion, load adjusting, and adaptation to non-critical failure. Huge data spread crosswise over numerous machines, need to parallelize. Moves the data, and gives booking, adaptation to non-critical failure. A writing survey on the MapReduce programming in different areas has completed in this paper. An examination course has been distinguished by utilizing a writing audit.

APA, Harvard, Vancouver, ISO, and other styles

9

Rokhman, Nur, and Amelia Nursanti. "The MapReduce Model on Cascading Platform for Frequent Itemset Mining." IJCCS (Indonesian Journal of Computing and Cybernetics Systems) 12, no. 2 (July 31, 2018): 149. http://dx.doi.org/10.22146/ijccs.34102.

Full text

Abstract:

The implementation of parallel algorithms is very interesting research recently. Parallelism is very suitable to handle large-scale data processing. MapReduce is one of the parallel and distributed programming models. The implementation of parallel programming faces many difficulties. The Cascading gives easy scheme of Hadoop system which implements MapReduce model.Frequent itemsets are most often appear objects in a dataset. The Frequent Itemset Mining (FIM) requires complex computation. FIM is a complicated problem when implemented on large-scale data. This paper discusses the implementation of MapReduce model on Cascading for FIM. The experiment uses the Amazon dataset product co-purchasing network metadata.The experiment shows the fact that the simple mechanism of Cascading can be used to solve FIM problem. It gives time complexity O(n), more efficient than the nonparallel which has complexity O(n2/m).

APA, Harvard, Vancouver, ISO, and other styles

10

Wang, Changjian, Yuxing Peng, Mingxing Tang, Dongsheng Li, Shanshan Li, and Pengfei You. "An Efficient MapReduce Computing Model for Imprecise Applications." International Journal of Web Services Research 13, no. 3 (July 2016): 46–63. http://dx.doi.org/10.4018/ijwsr.2016070103.

Full text

Abstract:

Optimizing the Map process is important for the improvement of the MapReduce performance. Many efforts have been devoted into the problem to design more efficient scheduling strategies. However, there exists a kind of MapReduce applications, named imprecise applications, where the imprecise results based on part of map tasks can satisfy the requirements of imprecise applications and thus the job processes can be completed when enough map tasks are processed. According to the feature of imprecise applications, the authors propose an improved MapReduce model, named MapCheckReduce, which can terminate the map process when the requirements of an imprecise application is satisfied. Compared to MapReduce, a Check mechanism and a set of extended programming interfaces are added to MapCheckReduce. The Check mechanism receives and analyzes messages submitted by completed map tasks and then determines whether to terminate the map phase according to the analysis results. The programming interfaces are used by the programmers to define the termination conditions of the map process. A data-prefetching mechanism is designed and implemented in MapCheckReduce which can improve the performance of MapCheckReduce effectively. The MapCheckReduce prototype has been implemented and experiment results verify the feasibility and effectiveness of MapCheckReduce.

APA, Harvard, Vancouver, ISO, and other styles

11

Tahsir Ahmed Munna, Md, Shaikh Muhammad Allayear, Mirza Mohtashim Alam, Sheikh Shah Mohammad Motiur Rahman, Md Samadur Rahman, and M. Mesbahuddin Sarker. "Simplified Mapreduce Mechanism for Large Scale Data Processing." International Journal of Engineering & Technology 7, no. 3.8 (July 7, 2018): 16. http://dx.doi.org/10.14419/ijet.v7i3.8.15211.

Full text

Abstract:

MapReduce has become a popular programming model for processing and running large-scale data sets with a parallel, distributed paradigm on a cluster. Hadoop MapReduce is needed especially for large scale data like big data processing. In this paper, we work to modify the Hadoop MapReduce Algorithm and implement it to reduce processing time.

APA, Harvard, Vancouver, ISO, and other styles

12

Sun, Han Lin. "An Improved MapReduce Model for Computation-Intensive Task." Advanced Materials Research 756-759 (September 2013): 1701–5. http://dx.doi.org/10.4028/www.scientific.net/amr.756-759.1701.

Full text

Abstract:

MapReduce is a widely adopted parallel programming model. The standard MapReduce model is designed for data-intensive processing. However, some machine learning algorithms are computation-intensive and time-consuming tasks which process the same data set repeatedly. In this paper, we proposed an improved MapReduce model for computation-intensive algorithms. The model is constructed from a service combination perspective. In the model, the whole task is divided into lots of subtasks taking account into the algorithms parameters, and the datagram with acknowledgement mechanism is used as the communication channel among cluster workers. We took the multifractal detrended fluctuation analysis algorithm as an example to demonstrate the model.

APA, Harvard, Vancouver, ISO, and other styles

13

Zhang, Weidong, Boxin He, Yifeng Chen, and Qifei Zhang. "Correction to: GMR: graph-compatible MapReduce programming model." Multimedia Tools and Applications 78, no. 1 (October 17, 2017): 477. http://dx.doi.org/10.1007/s11042-017-5273-x.

Full text

APA, Harvard, Vancouver, ISO, and other styles

14

Zheng, Feifeng, Zhaojie Wang, Yinfeng Xu, and Ming Liu. "Heuristic Algorithms for MapReduce Scheduling Problem with Open-Map Task and Series-Reduce Tasks." Scientific Programming 2020 (July 15, 2020): 1–10. http://dx.doi.org/10.1155/2020/8810215.

Full text

Abstract:

Based on the classical MapReduce concept, we propose an extended MapReduce scheduling model. In the extended MapReduce scheduling problem, we assumed that each job contains an open-map task (the map task can be divided into multiple unparallel operations) and series-reduce tasks (each reduce task consists of only one operation). Different from the classical MapReduce scheduling problem, we also assume that all the operations cannot be processed in parallel, and the machine settings are unrelated machines. For solving the extended MapReduce scheduling problem, we establish a mixed-integer programming model with the minimum makespan as the objective function. We then propose a genetic algorithm, a simulated annealing algorithm, and an L-F algorithm to solve this problem. Numerical experiments show that L-F algorithm has better performance in solving this problem.

APA, Harvard, Vancouver, ISO, and other styles

15

Kavitha, C., S. R. Srividhya, Wen-Cheng Lai, and Vinodhini Mani. "IMapC: Inner MAPping Combiner to Enhance the Performance of MapReduce in Hadoop." Electronics 11, no. 10 (May 17, 2022): 1599. http://dx.doi.org/10.3390/electronics11101599.

Full text

Abstract:

Hadoop is a framework for storing and processing huge amounts of data. With HDFS, large data sets can be managed on commodity hardware. MapReduce is a programming model for processing vast amounts of data in parallel. Mapping and reducing can be performed by using the MapReduce programming framework. A very large amount of data is transferred from Mapper to Reducer without any filtering or recursion, resulting in overdrawn bandwidth. In this paper, we introduce an algorithm called Inner MAPping Combiner (IMapC) for the map phase. This algorithm in the Mapper combines the values of recurring keys. In order to test the efficiency of the algorithm, different approaches were tested. According to the test, MapReduce programs that are implemented with the Default Combiner (DC) of IMapC will be 70% more efficient than those that are implemented without one. To make computations significantly faster, this work can be combined with MapReduce.

APA, Harvard, Vancouver, ISO, and other styles

16

Wei, Fang, Pan Wubin, and Cui Zhiming. "View of MapReduce: Programming model, methods, and its applications." IETE Technical Review 29, no. 5 (2012): 380. http://dx.doi.org/10.4103/0256-4602.103168.

Full text

APA, Harvard, Vancouver, ISO, and other styles

17

Liu, Hanpeng, Wuqi Gao, and Junmin Luo. "Research on Intelligentization of Cloud Computing Programs Based on Self-awareness." International Journal of Advanced Network, Monitoring and Controls 8, no. 2 (June 1, 2023): 89–98. http://dx.doi.org/10.2478/ijanmc-2023-0060.

Full text

Abstract:

Abstract Through the research of MapReduce programming framework of cloud computing, the current MapReduce program only solves specific problems, and there is no design experience or design feature summary of MapReduce program, let alone formal description and experience inheritance and application of knowledge base. In order to solve the problem of intelligent cloud computing program, a general MapReduce program generation method is designed. This paper proposes the architecture of intelligent cloud computing by studying AORBCO model and combining cloud computing technology. According to the behavior control mechanism in AORBCO model, a program generation method of MapReduce in intelligent cloud computing is proposed. This method will extract entity information in input data set and entity information in knowledge base in intelligent cloud computing for similarity calculation, and extract the entity in the top order as key key-value pair information in intelligent cloud computing judgment data set. The data processing types are divided, and then aligned with each specific MapReduce capability, and the MapReduce program generation experiment is verified in the AORBCO model development platform. The experiment shows that the complexity of big data MapReduce program code is simplified, and the generated code execution efficiency is good.

APA, Harvard, Vancouver, ISO, and other styles

18

Al-Badarneh, Amer, Amr Mohammad, and Salah Harb. "A Survey on MapReduce Implementations." International Journal of Cloud Applications and Computing 6, no. 1 (January 2016): 59–87. http://dx.doi.org/10.4018/ijcac.2016010104.

Full text

Abstract:

A distinguished successful platform for parallel data processing MapReduce is attracting a significant momentum from both academia and industry as the volume of data to capture, transform, and analyse grows rapidly. Although MapReduce is used in many applications to analyse large scale data sets, there is still a lot of debate among scientists and researchers on its efficiency, performance, and usability to support more classes of applications. This survey presents a comprehensive review of various implementations of MapReduce framework. Initially the authors give an overview of MapReduce programming model. They then present a broad description of various technical aspects of the most successful implementations of MapReduce framework reported in the literature and discuss their main strengths and weaknesses. Finally, the authors conclude by introducing a comparison between MapReduce implementations and discuss open issues and challenges on enhancing MapReduce.

APA, Harvard, Vancouver, ISO, and other styles

19

Amshakala, K., R. Nedunchezhian, and M. Rajalakshmi. "Extracting Functional Dependencies in Large Datasets Using MapReduce Model." International Journal of Intelligent Information Technologies 10, no. 3 (July 2014): 19–35. http://dx.doi.org/10.4018/ijiit.2014070102.

Full text

Abstract:

Over the last few years, data are generated in large volume at a faster rate and there has been a remarkable growth in the need for large scale data processing systems. As data grows larger in size, data quality is compromised. Functional dependencies representing semantic constraints in data are important for data quality assessment. Executing functional dependency discovery algorithms on a single computer is hard and laborious with large data sets. MapReduce provides an enabling technology for large scale data processing. The open-source Hadoop implementation of MapReduce has provided researchers a powerful tool for tackling large-data problems in a distributed manner. The objective of this study is to extract functional dependencies between attributes from large datasets using MapReduce programming model. Attribute entropy is used to measure the inter attribute correlations, and exploited to discover functional dependencies hidden in the data.

APA, Harvard, Vancouver, ISO, and other styles

20

Sontakke, Vaishali, and Dayananda R. B. "Memory aware optimized Hadoop MapReduce model in cloud computing environment." IAES International Journal of Artificial Intelligence (IJ-AI) 12, no. 3 (September 1, 2023): 1270. http://dx.doi.org/10.11591/ijai.v12.i3.pp1270-1280.

Full text

Abstract:

In the last decade, data analysis has become one of the popular tasks due to enormous growth in data every minute through different applications and instruments. MapReduce is the most popular programming model for data processing. Hadoop constitutes two basic models i.e., Hadoop file system (HDFS) and MapReduce, Hadoop is used for processing a huge amount of data whereas MapReduce is used for data processing. Hadoop MapReduce is one of the best platforms for processing huge data in an efficient manner such as processing web logs data. However, existing model This research work proposes memory aware optimized Hadoop MapReduce (MA-OHMR). MA-OHMR is developed considering memory as the constraint and prioritizes memory allocation and revocation in mapping, shuffling, and reducing, this further enhances the job of mapping and reducing. Optimal memory management and I/O operation are carried out to use the resource inefficiently manner. The model utilizes the global memory management to avoid garbage collection and MA-OHMR is optimized on the makespan front to reduce the same. MA-OHMR is evaluated considering two datasets i.e., simple workload of Wikipedia dataset and complex workload of sensor dataset considering makespan and cost as an evaluation parameter.

APA, Harvard, Vancouver, ISO, and other styles

21

Meng, Jian Liang, and Da Wei Li. "Improve and Optimize Query Recommendation System by MST Algorithm and its MapReduce Implementation." Applied Mechanics and Materials 701-702 (December 2014): 50–53. http://dx.doi.org/10.4028/www.scientific.net/amm.701-702.50.

Full text

Abstract:

Query recommendation as an important tool to enhance the user search efficiency has gradually become a hotspot. In the context of big data, using the MapReduce programming model, combined with distributed minimum spanning tree algorithm, a parallel query recommended method based on MapReduce was proposed in this paper. The final results show that the efficiency of query recommendation was greatly improved through parallel computing.

APA, Harvard, Vancouver, ISO, and other styles

22

Li, Ren, Haibo Hu, Heng Li, Yunsong Wu, and Jianxi Yang. "MapReduce Parallel Programming Model: A State-of-the-Art Survey." International Journal of Parallel Programming 44, no. 4 (October 29, 2015): 832–66. http://dx.doi.org/10.1007/s10766-015-0395-0.

Full text

APA, Harvard, Vancouver, ISO, and other styles

23

Jing, Weipeng, Danyu Tong, Yangang Wang, Jingyuan Wang, Yaqiu Liu, and Peng Zhao. "MaMR: High-performance MapReduce programming model for material cloud applications." Computer Physics Communications 211 (February 2017): 79–87. http://dx.doi.org/10.1016/j.cpc.2016.07.015.

Full text

APA, Harvard, Vancouver, ISO, and other styles

24

QIN, Jun, Yanyan SONG, and Ping ZONG. "Study of Task Scheduling Strategy based on Trustworthiness." International Journal of Distributed and Parallel systems 12, no. 05 (September 30, 2021): 01–09. http://dx.doi.org/10.5121/ijdps.2021.12501.

Full text

Abstract:

MapReduce is a distributed computing model for cloud computing to process massive data. It simplifies the writing of distributed parallel programs. For the fault-tolerant technology in the MapReduce programming model, tasks may be allocated to nodes with low reliability. It causes the task to be reexecuted, wasting time and resources. This paper proposes a reliability task scheduling strategy with a failure recovery mechanism, evaluates the trustworthiness of resource nodes in the cloud environment and builds a trustworthiness model. By using the simulation platform CloudSim, the stability of the task scheduling algorithm and scheduling model are verified in this paper.

APA, Harvard, Vancouver, ISO, and other styles

25

Park, Jong-Hyuk, Hwa-Young Jeong, Young-Sik Jeong, and Min Choi. "REST-MapReduce: An Integrated Interface but Differentiated Service." Journal of Applied Mathematics 2014 (2014): 1–10. http://dx.doi.org/10.1155/2014/170723.

Full text

Abstract:

With the fast deployment of cloud computing, MapReduce architectures are becoming the major technologies for mobile cloud computing. The concept of MapReduce was first introduced as a novel programming model and implementation for a large set of computing devices. In this research, we propose a novel concept of REST-MapReduce, enabling users to use only the REST interface without using the MapReduce architecture. This approach provides a higher level of abstraction by integration of the two types of access interface, REST API and MapReduce. The motivation of this research stems from the slower response time for accessing simple RDBMS on Hadoop than direct access to RDMBS. This is because there is overhead to job scheduling, initiating, starting, tracking, and management during MapReduce-based parallel execution. Therefore, we provide a good performance for REST Open API service and for MapReduce, respectively. This is very useful for constructing REST Open API services on Hadoop hosting services, for example, Amazon AWS (Macdonald, 2005) or IBM Smart Cloud. For evaluating performance of our REST-MapReduce framework, we conducted experiments with Jersey REST web server and Hadoop. Experimental result shows that our approach outperforms conventional approaches.

APA, Harvard, Vancouver, ISO, and other styles

26

Esposito, Christian, and Massimo Ficco. "Recent Developments on Security and Reliability in Large-Scale Data Processing with MapReduce." International Journal of Data Warehousing and Mining 12, no. 1 (January 2016): 49–68. http://dx.doi.org/10.4018/ijdwm.2016010104.

Full text

Abstract:

The demand to access to a large volume of data, distributed across hundreds or thousands of machines, has opened new opportunities in commerce, science, and computing applications. MapReduce is a paradigm that offers a programming model and an associated implementation for processing massive datasets in a parallel fashion, by using non-dedicated distributed computing hardware. It has been successfully adopted in several academic and industrial projects for Big Data Analytics. However, since such analytics is increasingly demanded within the context of mission-critical applications, security and reliability in MapReduce frameworks are strongly required in order to manage sensible information, and to obtain the right answer at the right time. In this paper, the authors present the main implementation of the MapReduce programming paradigm, provided by Apache with the name of Hadoop. They illustrate the security and reliability concerns in the context of a large-scale data processing infrastructure. They review the available solutions, and their limitations to support security and reliability within the context MapReduce frameworks. The authors conclude by describing the undergoing evolution of such solutions, and the possible issues for improvements, which could be challenging research opportunities for academic researchers.

APA, Harvard, Vancouver, ISO, and other styles

27

Khudhair, Muslim Mohsin, Adil AL-Rammahi, and Furkan Rabee. "An innovativefractal architecture model for implementing MapReduce in an open multiprocessing parallel environment." Indonesian Journal of Electrical Engineering and Computer Science 30, no. 2 (May 1, 2023): 1059. http://dx.doi.org/10.11591/ijeecs.v30.i2.pp1059-1067.

Full text

Abstract:

One of the infrastructure applications that cloud computing offers as a service is parallel data processing. MapReduce is a type of parallel processing used more and more by data-intensive applications in cloud computing environments. MapReduce is based on a strategy called "divide and conquer," which uses regular computers, also called "nodes," to do processing in parallel. This paper looks at how open multiprocessing (OpenMP), the best shared-memory parallel programming model for high-performance computing, can be used with the proposed fractal network model in the MapReduce application. A well-known model, the cube, is used to compare the fractal network model and its work. Where experiments demonstrated that the fractal model is preferable to the cube model. The fractal model achieved an average speedup of 2.7 and an efficiency rate of 67.7%. In contrast, the cube model could only reach an average speedup of 2.5 and an efficiency rate of 60.4%.

APA, Harvard, Vancouver, ISO, and other styles

28

Natesan, P., V. E. Sathishkumar, Sandeep Kumar Mathivanan, Maheshwari Venkatasen, Prabhu Jayagopal, and Shaikh Muhammad Allayear. "A Distributed Framework for Predictive Analytics Using Big Data and MapReduce Parallel Programming." Mathematical Problems in Engineering 2023 (February 1, 2023): 1–10. http://dx.doi.org/10.1155/2023/6048891.

Full text

Abstract:

With the advancement of Internet technologies and the rapid increase of World Wide Web applications, there has been tremendous growth in the volume of digital data. This takes the digital world into a new era of big data. Various existing data processing technologies are not consistent and scalable in handling the complexity as well as the large-size datasets. Recently, there are many distributed data processing, and programming models have been proposed and implemented to handle big data applications. The open-source-implemented MapReduce programming model in Apache Hadoop is the foremost model for data exhaustive and also computational-intensive applications due to its inherent characteristics of scalability, fault tolerance, and simplicity. In this research article, a new approach for the prediction of target labels in big data applications is developed using a multiple linear regression algorithm and MapReduce programming model, named as MR-MLR. This approach promises optimum values for MAE, RMSE, and determination coefficient (R2) and thus shows its effectiveness in predictions in big data applications.

APA, Harvard, Vancouver, ISO, and other styles

29

CHEN, Jirong, and Jiajin LE. "Programming model based on MapReduce for importing big table into HDFS." Journal of Computer Applications 33, no. 9 (November 7, 2013): 2486–89. http://dx.doi.org/10.3724/sp.j.1087.2013.02486.

Full text

APA, Harvard, Vancouver, ISO, and other styles

30

Zhang, Fan, and Qutaibah M. Malluhi. "A flexible and concurrent MapReduce programming model for shared-data applications." Qatar Foundation Annual Research Forum Proceedings, no. 2012 (October 2012): CSO10. http://dx.doi.org/10.5339/qfarf.2012.cso10.

Full text

APA, Harvard, Vancouver, ISO, and other styles

31

Gao, Yufei, Yanjie Zhou, Bing Zhou, Lei Shi, and Jiacai Zhang. "Handling Data Skew in MapReduce Cluster by Using Partition Tuning." Journal of Healthcare Engineering 2017 (2017): 1–12. http://dx.doi.org/10.1155/2017/1425102.

Full text

Abstract:

The healthcare industry has generated large amounts of data, and analyzing these has emerged as an important problem in recent years. The MapReduce programming model has been successfully used for big data analytics. However, data skew invariably occurs in big data analytics and seriously affects efficiency. To overcome the data skew problem in MapReduce, we have in the past proposed a data processing algorithm called Partition Tuning-based Skew Handling (PTSH). In comparison with the one-stage partitioning strategy used in the traditional MapReduce model, PTSH uses a two-stage strategy and the partition tuning method to disperse key-value pairs in virtual partitions and recombines each partition in case of data skew. The robustness and efficiency of the proposed algorithm were tested on a wide variety of simulated datasets and real healthcare datasets. The results showed that PTSH algorithm can handle data skew in MapReduce efficiently and improve the performance of MapReduce jobs in comparison with the native Hadoop, Closer, and locality-aware and fairness-aware key partitioning (LEEN). We also found that the time needed for rule extraction can be reduced significantly by adopting the PTSH algorithm, since it is more suitable for association rule mining (ARM) on healthcare data.

APA, Harvard, Vancouver, ISO, and other styles

32

Khudhair, Muslim Mohsin, Furkan Rabee, and Adil AL_Rammahi. "New efficient fractal models for MapReduce in OpenMP parallel environment." Bulletin of Electrical Engineering and Informatics 12, no. 4 (August 1, 2023): 2313–27. http://dx.doi.org/10.11591/beei.v12i4.4977.

Full text

Abstract:

Parallel data processing is one of the specific infrastructure applications categorized as a service provided by cloud computing. In cloud computing environments, data-intensive applications increasingly use the parallel processing paradigm known as MapReduce. MapReduce is based on a strategy called "divide and conquer," which uses ordinary computers, also called "nodes," to do processing in parallel. This paper looks at how open multiprocessing (OpenMP), the best shared-memory parallel programming model for high-performance computing, can be used in the MapReduce application using proposed fractal network models. Two fractal network models are offered, and their work is compared with a well-known network model, the hypercube. The first fractal network model achieved an average speedup of 3.239 times while an efficiency ranged from 73-95%. In the second model of the network, the speedup got to 3.236 times while keeping an efficiency of 70-92%. Furthermore, the path-finding algorithm employed in the recommended fractal network models remarkably identified all paths and calculated the shortest and longest routes.

APA, Harvard, Vancouver, ISO, and other styles

33

Khudhair, Muslim Mohsin, Furkan Rabee, and Adil AL_Rammahi. "New efficient fractal models for MapReduce in OpenMP parallel environment." Bulletin of Electrical Engineering and Informatics 12, no. 4 (August 1, 2023): 2313–27. http://dx.doi.org/10.11591/eei.v12i4.4977.

Full text

Abstract:

Parallel data processing is one of the specific infrastructure applications categorized as a service provided by cloud computing. In cloud computing environments, data-intensive applications increasingly use the parallel processing paradigm known as MapReduce. MapReduce is based on a strategy called "divide and conquer," which uses ordinary computers, also called "nodes," to do processing in parallel. This paper looks at how open multiprocessing (OpenMP), the best shared-memory parallel programming model for high-performance computing, can be used in the MapReduce application using proposed fractal network models. Two fractal network models are offered, and their work is compared with a well-known network model, the hypercube. The first fractal network model achieved an average speedup of 3.239 times while an efficiency ranged from 73-95%. In the second model of the network, the speedup got to 3.236 times while keeping an efficiency of 70-92%. Furthermore, the path-finding algorithm employed in the recommended fractal network models remarkably identified all paths and calculated the shortest and longest routes.

APA, Harvard, Vancouver, ISO, and other styles

34

Wang, Xiao Feng. "The Application of Hadoop in the Campus Cloud Computing System." Applied Mechanics and Materials 543-547 (March 2014): 3092–95. http://dx.doi.org/10.4028/www.scientific.net/amm.543-547.3092.

Full text

Abstract:

Based on the theory of cloud computing, this paper uses Hadoop distributed computing framework and the MapReduce programming model, designs and implements a campus cloud computing system for processing huge amounts of data. The system uses a three-layer architecture, has the flexibility to expand the scale, low development cost and ease of operation, reduces the difficulty of parallel programming and has the ability to efficiently handle massive data analysis and processing.

APA, Harvard, Vancouver, ISO, and other styles

35

He, Yuheng, Jin Qian, Juanjie Zhang, and Renzhe Zhang. "Word frequency statistics based on MapReduce on serverless platforms." Applied and Computational Engineering 68, no. 1 (July 31, 2024): 356–67. http://dx.doi.org/10.54254/2755-2721/68/20241536.

Full text

Abstract:

This paper investigates the application of serverless computing in conjunction with the MapReduce framework, particularly in machine learning (ML) tasks. The MapReduce programming model has been widely used to process large-scale datasets by simplifying parallel and distributed data processing. This study explores how the combination of these two technologies can provide more efficient and cost-effective ML solutions. Through a detailed analysis of serverless environments and the MapReduce framework, this paper shows how the combination can advance the fields of cloud computing and machine learning. The experimental part includes the implementation of Map-Reduce model on a serverless platform, exploring the impact of different parameter settings on performance and improving efficiency by optimizing the data processing flow. In addition, the paper analyzes the use of memory and CPU resources and derives the relationship between dataset size, memory consumption and processor configuration and execution time. Through these experiments and analyses, this paper provides an empirical basis and theoretical support for the optimization of cloud computing frameworks.

APA, Harvard, Vancouver, ISO, and other styles

36

Wang, Peng, Jia Nan Wang, Ji Ci Ba, and Yu Tan. "Treatment and Research of Massive Data Mining Based on Cloud Computing." Advanced Materials Research 765-767 (September 2013): 941–44. http://dx.doi.org/10.4028/www.scientific.net/amr.765-767.941.

Full text

Abstract:

This paper introduces SPRINT algorithm optimized in the Hadoop core framework. Combing the data mining process, we will study the cloud computing in the MapReduce programming model, then improve and optimize the SPRINT algorithm in conjunction with the mode, transplant the optimized algorithm to Hadoop platform for distributed data processing.

APA, Harvard, Vancouver, ISO, and other styles

37

Chandra Sekhar Reddy, L., and Dr D. Murali. "YouTube: big data analytics using Hadoop and map reduce." International Journal of Engineering & Technology 7, no. 3.29 (August 24, 2018): 12. http://dx.doi.org/10.14419/ijet.v7i3.29.18451.

Full text

Abstract:

We live today in a digital world a tremendous amount of data is generated by each digital service we use. This vast amount of data generated is called Big Data. According to Wikipedia, Big Data is a word for large data sets or compositions that the traditional data monitoring application software is pitiful to compress [5]. Extensive data cannot be used to receive data, store data, analyse data, search, share, transfer, view, consult, and update and maintain the confidentiality of information. Google's streaming services, YouTube, are one of the best examples of services that produce a massive amount of data in a brief period. Data extraction of a significant amount of data is done using Hadoop and MapReduce to measure performance. Hadoop is a system that offers consistent memory. Storage is provided by HDFS (Hadoop Distributed File System) and MapReduce analysis. MapReduce is a programming model and a corresponding implementation for processing large data sets. This article presents the analysis of Big Data on YouTube using the Hadoop and MapReduce techniques.

APA, Harvard, Vancouver, ISO, and other styles

38

Muhammad, Sharafadeen, Ibrahim Kabiru Dahiru, Ahmad Abubakar, and Muhammad Sanusi Ibrahim. "MODELING OF SYSTEMS UNDER CLOUD ENVIRONMENT." ASEAN Engineering Journal 11, no. 3 (April 21, 2021): 190–98. http://dx.doi.org/10.11113/aej.v11.17054.

Full text

Abstract:

The emergence of large amount of data requires an efficient means of processing and storage facilities. Cloud computing provides an effective solution; MapReduce programming paradigm has the ability to handle such data by implementing Hadoop, but came up with some conflicting challenges in terms of Service Level Agreement (SLA) between major stakeholders. This paper focuses on coming up with a MapReduce model through system identification in order to address the requirement of the service time to meet-up the SLA within the limit of defined threshold in the presence of uncertainties in the system. A second order nonlinear model was obtained, which shows a good representation of the real system and could be used to develop control laws on the real system.

APA, Harvard, Vancouver, ISO, and other styles

39

Gévay, Gábor E., Juan Soto, and Volker Markl. "Handling Iterations in Distributed Dataflow Systems." ACM Computing Surveys 54, no. 9 (December 31, 2022): 1–38. http://dx.doi.org/10.1145/3477602.

Full text

Abstract:

Over the past decade, distributed dataflow systems (DDS) have become a standard technology. In these systems, users write programs in restricted dataflow programming models, such as MapReduce, which enable them to scale out program execution to a shared-nothing cluster of machines. Yet, there is no established consensus that prescribes how to extend these programming models to support iterative algorithms. In this survey, we review the research literature and identify how DDS handle control flow, such as iteration, from both the programming model and execution level perspectives. This survey will be of interest for both users and designers of DDS.

APA, Harvard, Vancouver, ISO, and other styles

40

Wu, Yao, Long Zheng, Brian Heilig, and Guang R. Gao. "HAMR: A dataflow-based real-time in-memory cluster computing engine." International Journal of High Performance Computing Applications 31, no. 5 (October 10, 2016): 361–74. http://dx.doi.org/10.1177/1094342016672080.

Full text

Abstract:

As the attention given to big data grows, cluster computing systems for distributed processing of large data sets become the mainstream and critical requirement in high performance distributed system research. One of the most successful systems is Hadoop, which uses MapReduce as a programming/execution model and takes disks as intermedia to process huge volumes of data. Spark, as an in-memory computing engine, can solve the iterative and interactive problems more efficiently. However, currently it is a consensus that they are not the final solutions to big data due to a MapReduce-like programming model, synchronous execution model and the constraint that only supports batch processing, and so on. A new solution, especially, a fundamental evolution is needed to bring big data solutions into a new era. In this paper, we introduce a new cluster computing system called HAMR which supports both batch and streaming processing. To achieve better performance, HAMR integrates high performance computing approaches, i.e. dataflow fundamental into a big data solution. With more specifications, HAMR is fully designed based on in-memory computing to reduce the unnecessary disk access overhead; task scheduling and memory management are in fine-grain manner to explore more parallelism; asynchronous execution improves efficiency of computation resource usage, and also makes workload balance across the whole cluster better. The experimental results show that HAMR can outperform Hadoop MapReduce and Spark by up to 19x and 7x respectively, in the same cluster environment. Furthermore, HAMR can handle scaling data size well beyond the capabilities of Spark.

APA, Harvard, Vancouver, ISO, and other styles

41

González-Vélez, Horacio, and Maryam Kontagora. "Performance evaluation of MapReduce using full virtualisation on a departmental cloud." International Journal of Applied Mathematics and Computer Science 21, no. 2 (June 1, 2011): 275–84. http://dx.doi.org/10.2478/v10006-011-0020-3.

Full text

Abstract:

Performance evaluation of MapReduce using full virtualisation on a departmental cloudThis work analyses the performance of Hadoop, an implementation of the MapReduce programming model for distributed parallel computing, executing on a virtualisation environment comprised of 1+16 nodes running the VMWare workstation software. A set of experiments using the standard Hadoop benchmarks has been designed in order to determine whether or not significant reductions in the execution time of computations are experienced when using Hadoop on this virtualisation platform on a departmental cloud. Our findings indicate that a significant decrease in computing times is observed under these conditions. They also highlight how overheads and virtualisation in a distributed environment hinder the possibility of achieving the maximum (peak) performance.

APA, Harvard, Vancouver, ISO, and other styles

42

Dong, Li Li, Yu Jie Zhu, and Xiang Zhang. "A Microblogging Opinion Leader Recognition Algorithm Based on MapReduce." Applied Mechanics and Materials 571-572 (June 2014): 410–15. http://dx.doi.org/10.4028/www.scientific.net/amm.571-572.410.

Full text

Abstract:

Related researches on the influence of microblogging users are only given users’ influence ranking, while cannot determine the problem that which user plays a guiding role in the dissemination of information in the microblogging network. This paper proposed a microblogging opinion leader recognition algorithm called LeadersRank based on personalized PageRank algorithm. On the basis of LeadersRank algorithm research, since the problem of that current microblogging information has been massive data, using the idea of MapReduce programming model to improve the LeadersRank algorithm, so that designed a LeadersRank distributed parallel algorithm based on MapReduce algorithm running in the cloud platform hadoop environment. Finally, experiments verified the effectiveness of the two methods, and made analysis of experimental results.

APA, Harvard, Vancouver, ISO, and other styles

43

Lathar, Pankaj, and K. G. Srinivasa. "A Study on the Performance and Scalability of Apache Flink Over Hadoop MapReduce." International Journal of Fog Computing 2, no. 1 (January 2019): 61–73. http://dx.doi.org/10.4018/ijfc.2019010103.

Full text

Abstract:

With the advancements in science and technology, data is being generated at a staggering rate. The raw data generated is generally of high value and may conceal important information with the potential to solve several real-world problems. In order to extract this information, the raw data available must be processed and analysed efficiently. It has however been observed, that such raw data is generated at a rate faster than it can be processed by traditional methods. This has led to the emergence of the popular parallel processing programming model – MapReduce. In this study, the authors perform a comparative analysis of two popular data processing engines – Apache Flink and Hadoop MapReduce. The analysis is based on the parameters of scalability, reliability and efficiency. The results reveal that Flink unambiguously outperformance Hadoop's MapReduce. Flink's edge over MapReduce can be attributed to following features – Active Memory Management, Dataflow Pipelining and an Inline Optimizer. It can be concluded that as the complexity and magnitude of real time raw data is continuously increasing, it is essential to explore newer platforms that are adequately and efficiently capable of processing such data.

APA, Harvard, Vancouver, ISO, and other styles

44

Anantharaman, Padmanathan, and H. V. Ramakrishan. "Data Mining Itemset of Big Data Using Pre-Processing Based on Mapreduce FrameWork with ETL Tools." APTIKOM Journal on Computer Science and Information Technologies 2, no. 2 (July 1, 2017): 57–62. http://dx.doi.org/10.11591/aptikom.j.csit.103.

Full text

Abstract:

As data volumes continue to grow, they quickly consume the capacity of data warehouses and application databases. Is your IT organization forced into costly upgrades to expensive databases and data warehouse hardware appliances and enormous amount of data is getting explored through Internet of Things (IoT) as technologies are advancing and people uses these technologies in day to day activities, this data is termed as Big Data having its characteristics and challenges. Frequent Itemset Mining algorithms are aimed to disclose frequent itemsets from transactional database but as the dataset size increases, it cannot be handled by traditional frequent itemset mining. MapReduce programming model solves the problem of large datasets but it has large communication cost which reduces execution efficiency. This proposed new pre-processed k-means technique applied on BigFIM algorithm. ClustBigFIM uses hybrid approach, clustering using k-means algorithm to generate Clusters from huge datasets and Apriori and Eclat to mine frequent itemsets from generated clusters using MapReduce programming model. Results shown that execution efficiency of ClustBigFIM algorithm is increased by applying k-means clustering algorithm before BigFIM algorithm as one of the pre-processing technique.

APA, Harvard, Vancouver, ISO, and other styles

45

Li, Jianjiang, Yajun Liu, Jian Pan, Peng Zhang, Wei Chen, and Lizhe Wang. "Map-Balance-Reduce: An improved parallel programming model for load balancing of MapReduce." Future Generation Computer Systems 105 (April 2020): 993–1001. http://dx.doi.org/10.1016/j.future.2017.03.013.

Full text

APA, Harvard, Vancouver, ISO, and other styles

46

Erritali, Mohammed, Abderrahim Beni-Hssane, Marouane Birjali, and Youness Madani. "An Approach of Semantic Similarity Measure between Documents Based on Big Data." International Journal of Electrical and Computer Engineering (IJECE) 6, no. 5 (October 1, 2016): 2454. http://dx.doi.org/10.11591/ijece.v6i5.10853.

Full text

Abstract:

Semantic indexing and document similarity is an important information retrieval system problem in Big Data with broad applications. In this paper, we investigate MapReduce programming model as a specific framework for managing distributed processing in a large of amount documents. Then we study the state of the art of different approaches for computing the similarity of documents. Finally, we propose our approach of semantic similarity measures using WordNet as an external network semantic resource. For evaluation, we compare the proposed approach with other approaches previously presented by using our new MapReduce algorithm. Experimental results review that our proposed approach outperforms the state of the art ones on running time performance and increases the measurement of semantic similarity.

APA, Harvard, Vancouver, ISO, and other styles

47

Erritali, Mohammed, Abderrahim Beni-Hssane, Marouane Birjali, and Youness Madani. "An Approach of Semantic Similarity Measure between Documents Based on Big Data." International Journal of Electrical and Computer Engineering (IJECE) 6, no. 5 (October 1, 2016): 2454. http://dx.doi.org/10.11591/ijece.v6i5.pp2454-2461.

Full text

Abstract:

Semantic indexing and document similarity is an important information retrieval system problem in Big Data with broad applications. In this paper, we investigate MapReduce programming model as a specific framework for managing distributed processing in a large of amount documents. Then we study the state of the art of different approaches for computing the similarity of documents. Finally, we propose our approach of semantic similarity measures using WordNet as an external network semantic resource. For evaluation, we compare the proposed approach with other approaches previously presented by using our new MapReduce algorithm. Experimental results review that our proposed approach outperforms the state of the art ones on running time performance and increases the measurement of semantic similarity.

APA, Harvard, Vancouver, ISO, and other styles

48

Kang, Minsu, and Sang-Hoon Cho. "RHadoop-based Algorithm Utilizing QR Factorization for Multiple Linear Regression Analysis." Korean Data Analysis Society 25, no. 1 (February 28, 2023): 99–113. http://dx.doi.org/10.37727/jkdas.2022.25.1.99.

Full text

Abstract:

RHadoop enables R users to perform big data analytics under R programming environment by integrating R with Hadoop that supports distributed storing and parallel processing of large-scaled data. This article proposes a RHadoop-based MapReduce programming model for estimating multiple linear regression models utilizing QR factorization. Our proposed algorithm accommodates the most common type of big data that has a vast number of data points with only a few hundred variables. For QR factorization over massive-scaled data, our algorithm employs DirectQR method proposed by Benson, Gleich, Demmel(2013); however, it does not necessitate its iterative steps to estimate regression coefficients. Through a comparative simulation study, our algorithm is compared with a MapReduce-based algorithm proposed in the previous studies. For generating realistic synthetic data, we utilize NYC(New York City) yellow taxi trip data reported to NYC Taxi and Limousine Commission. In the simulation, we measure estimation time and accuracy of each algorithm under several assumptions with respect to the strength of association between independent variables and the noise level of error terms in the regression models.

APA, Harvard, Vancouver, ISO, and other styles

49

Lorenzo, Jorge González, José Emilio Labra Gayo, and José María Álvarez Rodríguez. "A MapReduce Implementation of the Spreading Activation Algorithm for Processing Large Knowledge Bases Based on Semantic Networks." International Journal of Knowledge Society Research 3, no. 4 (October 2012): 47–56. http://dx.doi.org/10.4018/jksr.2012100105.

Full text

Abstract:

The emerging Web of Data as part of the Semantic Web initiative and the sheer mass of information now available make it possible the deployment of new services and applications based on the reuse of existing vocabularies and datasets. A huge amount of this information is published by governments and organizations using semantic web languages and formats such as RDF, implicit graph structures developed using W3C standard languages: RDF-Schema or OWL, but new flexible programming models to process and exploit this data are required. In that sense the use of algorithms such as Spreading Activation is growing in order to find relevant and related information in this new data realm. Nevertheless the efficient exploration of the large knowledge bases has not yet been resolved and that is why new paradigms are emerging to boost the definitive deployment of the Web of Data. This cornerstone is being addressed applying new programming models such as MapReduce in combination with old-fashioned techniques of Document and Information Retrieval. In this paper an implementation of the Spreading Activation technique based on the MapReduce programming model and the problems of applying this paradigm to graph-based structures are introduced. Finally, a concrete experiment with real data is presented to illustrate the algorithm performance and scalability.

APA, Harvard, Vancouver, ISO, and other styles

50

Zou, Hongpeng. "MapReduce Algorithm on a Serverless Platform." Highlights in Science, Engineering and Technology 94 (April 26, 2024): 604–10. http://dx.doi.org/10.54097/9xv7vx88.

Full text

Abstract:

Using the MapReduce (MR) programming technique, large-scale data processing tasks can be divided into tasks that are easier to manage and independent. The serverless implementation's performance is better than the non-serverless MR model when the model parameters are altered in the experiment. Optimizing the use of the MR model on the serverless platform can be achieved by taking into account the relationship between implementation efficiency and platform settings. Additionally, it can act as a source of inspiration for future serverless hardware support configurations. Also, the serverless platform demonstrates how it improves the effectiveness of resource utilization in machine learning training. The intended result can then be achieved by combining the results of these concurrently operating jobs on server clusters. This work adapts MR, a popular big data processing framework, to the serverless platform, emphasizing realization simulation principles and services, and then uses the results in a word count experiment. The experiment uses a word count of about eleven thousand words to evaluate how well MR is implemented on Alibaba Cloud. It is validated for execution time on the platform with varying Central Processing Unit (CPU) core counts, memory configurations, and worker counts. By testing several platform configurations, it is found that the memory configuration has very little impact on the model's execution time, while the size of the CPU core has a considerable impact on reaction time relative to the number of workers.

APA, Harvard, Vancouver, ISO, and other styles

Journal articles on the topic 'MapReduce programming model'

Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles