Journal articles on the topic 'Stream Processing System'
To see the other types of publications on this topic, follow the link: Stream Processing System.
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the top 50 journal articles for your research on the topic 'Stream Processing System.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Browse journal articles on a wide variety of disciplines and organise your bibliography correctly.
1
Shuiying Yu, Shuiying Yu, Yinting Zheng Shuiying Yu, Fan Zhang Yinting Zheng, Hanhua Chen Fan Zhang, and Hai Jin Hanhua Chen. "TriJoin: A Time-Efficient and Scalable Three-Way Distributed Stream Join System." 網際網路技術學刊 24, no. 2 (March 2023): 475–85. http://dx.doi.org/10.53106/160792642023032402024.
Full textAbstract:
<p>Stream join is one of the most fundamental operations in data stream processing applications. Existing distributed stream join systems can support efficient two-way join, which is a join operation between two streams. Based the two-way join, implementing a three-way join require to be split into double two-way joins, where the second two-way join needs to wait for the join result transmitted from the first two-way join. We show through experiments that such a design raises prohibitively high processing latency. To solve this problem, we propose TriJoin, a time-efficient three-way distributed stream join system. We design a symmetric wait-free structure by symmetrically partitioning tuples and reused join. TriJoin utilizes reused join to join each new tuple with the intermediate result of the other two streams and stored tuples locally. For a new tuple, TriJoin only joins it with the intermediate result to generate the final result without waiting, greatly reducing the processing latency. In TriJoin, we design two partitioning and storage schemes according to two different forms of three-way stream join. We implement TriJoin and conduct comprehensive experiments to evaluate the performance using real-world traces. Results show that TriJoin significantly reduces the processing latency by up to 68%, compared to existing designs.</p> <p> </p>
2
Shi, Peng, and Li Li. "Design of Network Analysis System Based on Stream Computing." Journal of Computational and Theoretical Nanoscience 14, no. 1 (January 1, 2017): 64–68. http://dx.doi.org/10.1166/jctn.2017.6125.
Full textAbstract:
The functions of the network analysis system include detection and analysis of network data stream. According to the results of the network analysis, we monitor the network accident and avoid the security risks. This can improve the network performance and increase the network availability. As the data flow in the network is constantly produced, the biggest characteristic of network analysis system is that it is a real-time system. Because of the high requirements of the network data analysis and network fault processing, the system requires very high processing efficiency of the real time data of network. Stream computing is a technique specifically for processing real-time data streams. Its idea is that the value of the data is reduced with the lapse of time, so as long as the data appearing, it must be processed as soon as possible. So we use the technology of stream computing to design network analysis system to meet the needs of real-time capability. Moreover, the stream computing framework has been widely welcomed in the field because of its good expansibility, ease of use and flexibility. In this paper, firstly, we introduce the characteristics of the data processing based on stream computing and the traditional data processing separately. We point out their difference and introduce the technique of stream computing. Then, we introduce the architecture of network analysis system designed base on the technique of stream computing. The architecture includes two main components that are logic processing layer and communication layer. We describe the characteristics of each component and functional characteristics in detail, and we introduce the system load balancing algorithm. Finally, by experiments, we verify the effectiveness of the system’s characteristics of dynamic expansion and load balancing.
3
Bernardelli de Moraes, Matheus, and André Leon Sampaio Gradvohl. "Evaluating the impact of a coordinated checkpointing in distributed data streams processing systems using discrete event simulation." Revista Brasileira de Computação Aplicada 12, no. 2 (May 19, 2020): 16–27. http://dx.doi.org/10.5335/rbca.v12i2.10295.
Full textAbstract:
Data Streams Processing systems process continuous flows of data under Quality of Service requirements. Data streams often contain critical information which requires real-time processing. To guarantee systems' dependability and avoid information loss, one must use a fault-tolerance strategy. However, there are several strategies available, and the proper evaluation of which mechanism is better for each system architecture is challenging, especially in large-scale distributed systems. In this paper, we propose a discrete simulation model for investigating the impacts of the Coordinated Checkpoint fault tolerance strategy imposes on Data Stream Processing Systems. Results show that this strategy critically affects stream processing in failure-prone situations due to an increase in latency up to 120% and information loss, reaching 95% of the processing window in the worst case.
4
Valeev, S. S., N. V. Kondratyeva, A. S. Kovtunenko, M. A. Timirov, and R. R. Karimov. "Distributed stream data processing system in multi-agent safety system of infrastructure objects." Information Technology and Nanotechnology, no. 2416 (2019): 324–31. http://dx.doi.org/10.18287/1613-0073-2019-2416-324-331.
Full textAbstract:
The solution of the problem of resource management in distributed computing systems of processing stream data in safety systems of distributed objects is considered. The tasks of streaming data processing in a multi-level multi-agent evacuation system in an infrastructure object are considered. The features of the mathematical model of a distributed stream data processing system are discussed.
5
Ye, Qian, and Minyan Lu. "s2p: Provenance Research for Stream Processing System." Applied Sciences 11, no. 12 (June 15, 2021): 5523. http://dx.doi.org/10.3390/app11125523.
Full textAbstract:
The main purpose of our provenance research for DSP (distributed stream processing) systems is to analyze abnormal results. Provenance for these systems is not nontrivial because of the ephemerality of stream data and instant data processing mode in modern DSP systems. Challenges include but are not limited to an optimization solution for avoiding excessive runtime overhead, reducing provenance-related data storage, and providing it in an easy-to-use fashion. Without any prior knowledge about which kinds of data may finally lead to the abnormal, we have to track all transformations in detail, which potentially causes hard system burden. This paper proposes s2p (Stream Process Provenance), which mainly consists of online provenance and offline provenance, to provide fine- and coarse-grained provenance in different precision. We base our design of s2p on the fact that, for a mature online DSP system, the abnormal results are rare, and the results that require a detailed analysis are even rarer. We also consider state transition in our provenance explanation. We implement s2p on Apache Flink named as s2p-flink and conduct three experiments to evaluate its scalability, efficiency, and overhead from end-to-end cost, throughput, and space overhead. Our evaluation shows that s2p-flink incurs a 13% to 32% cost overhead, 11% to 24% decline in throughput, and few additional space costs in the online provenance phase. Experiments also demonstrates the s2p-flink can scale well. A case study is presented to demonstrate the feasibility of the whole s2p solution.
6
Al Jawarneh, Isam Mashhour, Paolo Bellavista, Antonio Corradi, Luca Foschini, and Rebecca Montanari. "QoS-Aware Approximate Query Processing for Smart Cities Spatial Data Streams." Sensors 21, no. 12 (June 17, 2021): 4160. http://dx.doi.org/10.3390/s21124160.
Full textAbstract:
Large amounts of georeferenced data streams arrive daily to stream processing systems. This is attributable to the overabundance of affordable IoT devices. In addition, interested practitioners desire to exploit Internet of Things (IoT) data streams for strategic decision-making purposes. However, mobility data are highly skewed and their arrival rates fluctuate. This nature poses an extra challenge on data stream processing systems, which are required in order to achieve pre-specified latency and accuracy goals. In this paper, we propose ApproxSSPS, which is a system for approximate processing of geo-referenced mobility data, at scale with quality of service guarantees. We focus on stateful aggregations (e.g., means, counts) and top-N queries. ApproxSSPS features a controller that interactively learns the latency statistics and calculates proper sampling rates to meet latency or/and accuracy targets. An overarching trait of ApproxSSPS is its ability to strike a plausible balance between latency and accuracy targets. We evaluate ApproxSSPS on Apache Spark Structured Streaming with real mobility data. We also compared ApproxSSPS against a state-of-the-art online adaptive processing system. Our extensive experiments prove that ApproxSSPS can fulfill latency and accuracy targets with varying sets of parameter configurations and load intensities (i.e., transient peaks in data loads versus slow arriving streams). Moreover, our results show that ApproxSSPS outperforms the baseline counterpart by significant magnitudes. In short, ApproxSSPS is a novel spatial data stream processing system that can deliver real accurate results in a timely manner, by dynamically specifying the limits on data samples.
7
Li, Huiyong, Xiaofeng Wu, and Yanhong Wang. "Dynamic Performance Analysis of STEP System in Internet of Vehicles Based on Queuing Theory." Computational Intelligence and Neuroscience 2022 (April 10, 2022): 1–13. http://dx.doi.org/10.1155/2022/8322029.
Full textAbstract:
The Internet of vehicles (IoV) is an important research area of the intelligent transportation systems using Internet of things theory. The complex event processing technology is a basic issue for processing the data stream in IoV. In recent years, many researchers process the temporal and spatial data flow by complex event processing technology. Spatial Temporal Event Processing (STEP) is a complex event query language focusing on the temporal and spatial data flow in Internet of vehicles. There are four processing models of the event stream processing system based on the complex event query language: finite automata model, matching tree model, directed acyclic graph model, and Petri net model. In addition, the worst-case response time of the event stream processing system is an important indicator of evaluating the performance of the system. Firstly, this paper proposed a core algorithm of the temporal and spatial event stream processing program based on STEP by Petri net model. Secondly, we proposed a novel method to estimate the worst-case response time of the event stream processing system, which is based on stochastic Petri net and queuing theory. Finally, through the simulation experiment based on queuing theory, this paper proves that the data stream processing system based on STEP has good dynamic performance in processing the spatiotemporal data stream in Internet of vehicles.
8
Akanbi, Adeyinka, and Muthoni Masinde. "A Distributed Stream Processing Middleware Framework for Real-Time Analysis of Heterogeneous Data on Big Data Platform: Case of Environmental Monitoring." Sensors 20, no. 11 (June 3, 2020): 3166. http://dx.doi.org/10.3390/s20113166.
Full textAbstract:
In recent years, the application and wide adoption of Internet of Things (IoT)-based technologies have increased the proliferation of monitoring systems, which has consequently exponentially increased the amounts of heterogeneous data generated. Processing and analysing the massive amount of data produced is cumbersome and gradually moving from classical ‘batch’ processing—extract, transform, load (ETL) technique to real-time processing. For instance, in environmental monitoring and management domain, time-series data and historical dataset are crucial for prediction models. However, the environmental monitoring domain still utilises legacy systems, which complicates the real-time analysis of the essential data, integration with big data platforms and reliance on batch processing. Herein, as a solution, a distributed stream processing middleware framework for real-time analysis of heterogeneous environmental monitoring and management data is presented and tested on a cluster using open source technologies in a big data environment. The system ingests datasets from legacy systems and sensor data from heterogeneous automated weather systems irrespective of the data types to Apache Kafka topics using Kafka Connect APIs for processing by the Kafka streaming processing engine. The stream processing engine executes the predictive numerical models and algorithms represented in event processing (EP) languages for real-time analysis of the data streams. To prove the feasibility of the proposed framework, we implemented the system using a case study scenario of drought prediction and forecasting based on the Effective Drought Index (EDI) model. Firstly, we transform the predictive model into a form that could be executed by the streaming engine for real-time computing. Secondly, the model is applied to the ingested data streams and datasets to predict drought through persistent querying of the infinite streams to detect anomalies. As a conclusion of this study, a performance evaluation of the distributed stream processing middleware infrastructure is calculated to determine the real-time effectiveness of the framework.
9
Otten, Lambert. "Wetdry composting of organic municipal solid waste: current status in Canada." Canadian Journal of Civil Engineering 28, S1 (January 1, 2001): 124–30. http://dx.doi.org/10.1139/l00-072.
Full textAbstract:
Source separation of municipal solid waste into wet and dry streams is proving to be an attractive alternative in dealing with solid waste, and in achieving provincial and national waste diversion objectives. The system provides important flexibility in the number of waste streams, collection methods, collection frequency, and waste processing. In the past few years, experience has been obtained with two-, three-, and four-stream source separation and collection, composting of the organic waste fraction, and recycling of the valuable dry waste. The systems used in Guelph, Ontario, Lunenburg, Nova Scotia, and Caledon, Ontario, are presented. Public interest and participation has been high, especially when a two-stream, mandatory system is used. Thus, the City of Guelph has reported a 98% participation rate in its two-stream system which means that the public accepted the two-stream approach. Experience has shown that, as the number of streams increase, there is a greater chance of putting waste in the wrong stream. There is a strong demand for compost at a bulk price of about $30/ t FOB at the plant. The processing cost of the three plants varied from $50/t to $80/t of waste received without allowing for credits derived from extended landfill life or reduction in environmental impact.Key words: municipal solid waste, organic, source-separation, composting.
10
Park, Alfred J., Cheng-Hong Li, Ravi Nair, Nobuyuki Ohba, Uzi Shvadron, Ayal Zaks, and Eugen Schenfeld. "Towards flexible exascale stream processing system simulation." SIMULATION 88, no. 7 (August 9, 2011): 832–51. http://dx.doi.org/10.1177/0037549711412981.
Full text
11
EITER, THOMAS, PAUL OGRIS, and KONSTANTIN SCHEKOTIHIN. "A Distributed Approach to LARS Stream Reasoning (System paper)." Theory and Practice of Logic Programming 19, no. 5-6 (September 2019): 974–89. http://dx.doi.org/10.1017/s1471068419000309.
Full textAbstract:
AbstractStream reasoning systems are designed for complex decision-making from possibly infinite, dynamic streams of data. Modern approaches to stream reasoning are usually performing their computations using stand-alone solvers, which incrementally update their internal state and return results as the new portions of data streams are pushed. However, the performance of such approaches degrades quickly as the rates of the input data and the complexity of decision problems are growing. This problem was already recognized in the area of stream processing, where systems became distributed in order to allocate vast computing resources provided by clouds. In this paper we propose a distributed approach to stream reasoning that can efficiently split computations among different solvers communicating their results over data streams. Moreover, in order to increase the throughput of the distributed system, we suggest an interval-based semantics for the LARS language, which enables significant reductions of network traffic. Performed evaluations indicate that the distributed stream reasoning significantly outperforms existing stand-alone LARS solvers when the complexity of decision problems and the rate of incoming data are increasing.
12
Alzghoul, Ahmad. "Monitoring Big Data Streams Using Data Stream Management Systems: Industrial Needs, Challenges, and Improvements." Advances in Operations Research 2023 (June 27, 2023): 1–12. http://dx.doi.org/10.1155/2023/2596069.
Full textAbstract:
Real-time monitoring systems are important for industry since they allow for avoiding unplanned system stops and keeping system availability high. The technical requirements for such systems include being both scalable and online, as the amount of generated data is increasing with time. Therefore, monitoring systems must integrate tools that can manage and analyze the data streams. The data stream management system is a stream processing tool that has the ability to manage and support operations on data streams in real-time. Several researchers have proposed and tested real-time monitoring systems which have the ability to search big data streams. In this paper, the research works that discuss the analysis of online data streams for fault detection in industry are reviewed. Based on the literature analysis, the industrial needs and challenges of monitoring big data streams are presented. Furthermore, feasible suggestions for improving the real-time monitoring system are proposed.
13
Himmelbach, Marc, and Hans-Otto Karnath. "Dorsal and Ventral Stream Interaction: Contributions from Optic Ataxia." Journal of Cognitive Neuroscience 17, no. 4 (April 2005): 632–40. http://dx.doi.org/10.1162/0898929053467514.
Full textAbstract:
In monkeys and humans, two functionally specialized cortical streams of visual processing emanating from V1 have been proposed: a dorsal, action-related system and a ventral, perception-related pathway. Traditionally, a separate organization of the two streams is assumed; the extent of functional interaction is unknown. After lesions of the dorsal stream in patients with optic ataxia, it has recently been shown that the ventral perception-related system might contribute to visuo-motor processing if movements rely on remembered target positions. The ventral pathway thus seemed to participate in goal-directed movements, a function that previously has been assigned exclusively to the dorsal stream. We wondered whether different types of pointing movements are controlled by switching between two separated cortical pathways or whether a variable interaction of interconnected systems should be assumed. Our study investigated two acute stroke patients with optic ataxia following lesions of the dorsal stream in a delayed pointing task. The delays ranged from 0 to 10 sec. The patients' pointing error decreased in a linear manner with the length of time. The finding suggests a gradual change between dorsal and ventral control of reaching behavior, rather than a sudden switch between two separated cortical processing streams. Although our observations with two patients require further validation, the results suggest that the ventral and dorsal systems interact closely in the sensorimotor control of reaching behavior.
14
Cho, Wonhyeong, Myeong-Seon Gil, Mi-Jung Choi, and Yang-Sae Moon. "Storm-based distributed sampling system for multi-source stream environment." International Journal of Distributed Sensor Networks 14, no. 11 (November 2018): 155014771881269. http://dx.doi.org/10.1177/1550147718812698.
Full textAbstract:
As a large amount of data streams occur rapidly in many recent applications such as social network service, Internet of Things, and smart factory, sampling techniques have attracted many attentions to handle such data streams efficiently. In this article, we address the performance improvement of binary Bernoulli sampling in the multi-source stream environment. Binary Bernoulli sampling has the n:1 structure where n sites transmit data to 1 coordinator. However, as the number of sites increases or the input stream explosively increases, the binary Bernoulli sampling may cause a severe bottleneck in the coordinator. In addition, bidirectional communication over different networks among the coordinator and sites may incur excessive communication overhead. In this article, we propose a novel distributed processing model of binary Bernoulli sampling to solve these coordinator bottleneck and communication overhead problems. We first present a multiple-coordinator structure to solve the coordinator bottleneck. We then present a new sampling model with an integrated framework and shared memory to alleviate the communication overhead. To verify the effectiveness and scalability of the proposed model, we perform its actual implementation in Apache Storm, a real-time distributed stream processing system. Experimental results show that our Storm-based binary Bernoulli sampling improves performance by up to 1.8 times compared with the legacy method and maintains high performance even when the input stream largely increases. These results indicate that the proposed distributed processing model is an excellent approach that solves the performance degradation problem of binary Bernoulli sampling and verifies its superiority through the actual implementation on Apache Storm.
15
Lin, Edgar Chia Han. "Research on Sequence Query Processing Techniques over Data Streams." Applied Mechanics and Materials 284-287 (January 2013): 3507–11. http://dx.doi.org/10.4028/www.scientific.net/amm.284-287.3507.
Full textAbstract:
Due to the great progress of computer technology and mature development of network, more and more data are generated and distributed through the network, which is called data streams. During the last couple of years, a number of researchers have paid their attention to data stream management, which is different from the conventional database management. At present, the new type of data management system, called data stream management system (DSMS), has become one of the most popular research areas in data engineering field. Lots of research projects have made great progress in this area. Since the current DSMS does not support queries on sequence data, this project will study the issues related to two types of data. First, we will focus on the content filtering on single-attribute streams, such as sensor data. Second, we will focus on multi-attribute streams, such as video films. We will discuss the related issues such as how to build an efficient index for all queries of different streams and the corresponding query processing mechanisms.
16
Endler, Markus, Jean-Pierre Briot, Vitor P. de Almeida, Ruhan dos Reis, and Francisco Silva e Silva. "Stream-Based Reasoning for IoT Applications — Proposal of Architecture and Analysis of Challenges." International Journal of Semantic Computing 11, no. 03 (September 2017): 325–44. http://dx.doi.org/10.1142/s1793351x1740013x.
Full textAbstract:
As distributed IoT applications become larger and more complex, the pure processing of raw sensor and actuation data streams becomes impractical. Instead, data streams must be fused into tangible facts and these pieces of information must be combined with a background knowledge to infer new pieces of knowledge. And since many IoT applications require almost real-time reactivity to stimulus of the environment, such information inference process has to be performed in a continuous, on-line manner. This paper proposes a new semantic model for data stream processing and real-time reasoning based on the concepts of Semantic Stream and Fact Stream, as a natural extension of Complex Event Processing (CEP) and RDF (graph-based knowledge model). The main advantages of our approach are that: (a) it considers time as a key relation between pieces of information; (b) the processing of streams can be implemented using CEP; (c) it is general enough to be applied to any Data Stream Management System (DSMS). We describe a scenario about patients flux monitoring in a hospital as an example of prospective application. Last, we present challenges and prospects on using machine learning and induction algorithms to learn abstractions and reasoning rules from a continuous data stream.
17
Xiao, Fuyuan, and Masayoshi Aritsugi. "An Adaptive Parallel Processing Strategy for Complex Event Processing Systems over Data Streams in Wireless Sensor Networks." Sensors 18, no. 11 (November 2, 2018): 3732. http://dx.doi.org/10.3390/s18113732.
Full textAbstract:
Efficient matching of incoming events of data streams to persistent queries is fundamental to event stream processing systems in wireless sensor networks. These applications require dealing with high volume and continuous data streams with fast processing time on distributed complex event processing (CEP) systems. Therefore, a well-managed parallel processing technique is needed for improving the performance of the system. However, the specific properties of pattern operators in the CEP systems increase the difficulties of the parallel processing problem. To address these issues, a parallelization model and an adaptive parallel processing strategy are proposed for the complex event processing by introducing a histogram and utilizing the probability and queue theory. The proposed strategy can estimate the optimal event splitting policy, which can suit the most recent workload conditions such that the selected policy has the least expected waiting time for further processing of the arriving events. The proposed strategy can keep the CEP system running fast under the variation of the time window sizes of operators and the input rates of streams. Finally, the utility of our work is demonstrated through the experiments on the StreamBase system.
18
Lin, Edgar Chia Han. "Research on Multi-Attribute Sequence Query Processing Techniques over Data Streams." Applied Mechanics and Materials 513-517 (February 2014): 575–78. http://dx.doi.org/10.4028/www.scientific.net/amm.513-517.575.
Full textAbstract:
Due to the great progress of computer technology and mature development of network, more and more data are generated and distributed through the network, which is called data streams. During the last couple of years, a number of researchers have paid their attention to data stream management, which is different from the conventional database management. At present, the new type of data management system, called data stream management system (DSMS), has become one of the most popular research areas in data engineering field. Lots of research projects have made great progress in this area. Since the current DSMS does not support queries on sequence data, this paper, we will focus on multi-attribute streams, such as video films. We will discuss the related issues such as how to build an efficient index for all queries of different streams and the corresponding query processing mechanisms.
19
Xiao, Fuyuan, Cheng Zhan, Hong Lai, Li Tao, and Zhiguo Qu. "New parallel processing strategies in complex event processing systems with data streams." International Journal of Distributed Sensor Networks 13, no. 8 (August 2017): 155014771772862. http://dx.doi.org/10.1177/1550147717728626.
Full textAbstract:
Sensor network–based application has gained increasing attention where data streams gathered from distributed sensors need to be processed and analyzed with timely responses. Distributed complex event processing is an effective technology to handle these data streams by matching of incoming events to persistent pattern queries. Therefore, a well-managed parallel processing scheme is required to improve both system performance and the quality-of-service guarantees of the system. However, the specific properties of pattern operators increase the difficulties of implementing parallel processing. To address this issue, a new parallelization model and three parallel processing strategies are proposed for distributed complex event processing systems. The effects of temporal constraints, for example, sliding windows, are included in the new parallelization model to enable the processing load for the overlap between windows of a batch induced by each input event to be shared by the downstream machines to avoid events that may result in wrong decisions. The proposed parallel strategies can keep the complex event processing system working stably and continuously during the elapsed time. Finally, the application of our work is demonstrated using experiments on the StreamBase system regardless of the increased input rate of the stream or the increased time window size of the operator.
20
Broy, Manfred, and Claus Dendorfer. "Modelling operating system structures by timed stream processing functions." Journal of Functional Programming 2, no. 1 (January 1992): 1–21. http://dx.doi.org/10.1017/s0956796800000241.
Full textAbstract:
AbstractSome extensions of the basic formalism of stream processing functions are useful to specify complex structures such as operating systems. In this paper we give the foundations of higher order stream processing functions. These are functions which send and accept not only messages representing atomic data, but also complex elements such as functions. Some special notations are introduced for the specification and manipulation of such functions. A representation of time is outlined, which enables us to model time dependent behaviour. Finally, we demonstrate how characteristic operating system structures can be modelled by timed higher order stream processing functions.
21
Hanif, Muhammad, Choonhwa Lee, and Sumi Helal. "Predictive topology refinements in distributed stream processing system." PLOS ONE 15, no. 11 (November 5, 2020): e0240424. http://dx.doi.org/10.1371/journal.pone.0240424.
Full textAbstract:
Cloud computing has evolved the big data technologies to a consolidated paradigm with SPaaS (Streaming processing-as-a-service). With a number of enterprises offering cloud-based solutions to end-users and other small enterprises, there has been a boom in the volume of data, creating interest of both industry and academia in big data analytics, streaming applications, and social networking applications. With the companies shifting to cloud-based solutions as a service paradigm, the competition grows in the market. Good quality of service (QoS) is a must for the enterprises, as they strive to survive in a competitive environment. However, achieving reasonable QoS goals to meet SLA agreement cost-effectively is challenging due to variation in workload over time. This problem can be solved if the system has the ability to predict the workload for the near future. In this paper, we present a novel topology-refining scheme based on a workload prediction mechanism. Predictions are made through a model based on a combination of SVR, autoregressive, and moving average model with a feedback mechanism. Our streaming system is designed to increase the overall performance by making the topology refining robust to the incoming workload on the fly, while still being able to achieve QoS goals of SLA constraints. Apache Flink distributed processing engine is used as a testbed in the paper. The result shows that the prediction scheme works well for both workloads, i.e., synthetic as well as real traces of data.
22
Li, Yue-jie. "Data Stream of Wireless Sensor Networks Based on Deep Learning." International Journal of Online Engineering (iJOE) 12, no. 11 (November 24, 2016): 22. http://dx.doi.org/10.3991/ijoe.v12i11.6232.
Full textAbstract:
The sensor data in wireless sensor networks are continuously arriving in multiple, rapid, time varying, possibly unpredictable, unbounded streams, and no record of historical information is kept. These limitations make conventional Database Management Systems and their evolution unsuitable for streams. Thereby there is a need to build a complete Data Streaming Management System (DSMS), which could process streams and perform dynamic continuous query processing. In this paper, a framework for Adaptive Distributed Data Streaming Management System (ADDSMS) is presented, which operates as streams control interface between arrays of distributed data stream sources and end-user clients who access and analyze these streams. Simulation results show that the proposed method can thus improve overall system performance substantially.
23
Speechley, W. J., C. B. Murray, R. M. McKay, M. T. Munz, and E. T. C. Ngan. "A failure of conflict to modulate dual-stream processing may underlie the formation and maintenance of delusions." European Psychiatry 25, no. 2 (March 2010): 80–86. http://dx.doi.org/10.1016/j.eurpsy.2009.05.012.
Full textAbstract:
AbstractBackgroundDual-stream information processing proposes that reasoning is composed of two interacting processes: a fast, intuitive system (Stream 1) and a slower, more logical process (Stream 2). In non-patient controls, divergence of these streams may result in the experience of conflict, modulating decision-making towards Stream 2, and initiating a more thorough examination of the available evidence. In delusional schizophrenia patients, a failure of conflict to modulate decision-making towards Stream 2 may reduce the influence of contradictory evidence, resulting in a failure to correct erroneous beliefs.MethodDelusional schizophrenia patients and non-patient controls completed a deductive reasoning task requiring logical validity judgments of two-part conditional statements. Half of the statements were characterized by a conflict between logical validity (Stream 2) and content believability (Stream 1).ResultsPatients were significantly worse than controls in determining the logical validity of both conflict and non-conflict conditional statements. This between groups difference was significantly greater for the conflict condition.ConclusionsThe results are consistent with the hypothesis that delusional schizophrenia patients fail to use conflict to modulate towards Stream 2 when the two streams of reasoning arrive at incompatible judgments. This finding provides encouraging preliminary support for the Dual-Stream Modulation Failure model of delusion formation and maintenance.
24
Maison, Rafal, and Maciej Zakrzewicz. "Content-based load shedding in multimedia data stream management system." Foundations of Computing and Decision Sciences 37, no. 2 (October 1, 2012): 79–95. http://dx.doi.org/10.2478/v10209-011-0007-8.
Full textAbstract:
Abstract.Overload management has become very important in public safety systems that analyse high performance multimedia data streams, especially in the case of detection of terrorist and criminal dangers. Efficient overload management improves the accuracy of automatic identification of persons suspected of terrorist or criminal activity without requiring interaction with them. We argue that in order to improve the quality of multimedia data stream processing in the public safety arena, the innovative concept of a Multimedia Data Stream Management System (MMDSMS) using load-shedding techniques should be introduced into the infrastructure to monitor and optimize the execution of multimedia data stream queries. In this paper, we present a novel content-centered load shedding framework, based on searching and matching algorithms, for analysing video tuples arriving within multimedia data streams. The framework tracks and registers all symptoms of overload, and either prevents overload before it occurs, or minimizes its effects. We have extended our Continuous Query Language (CQL) syntax to enable this load shedding technique. The effectiveness of the framework has been verified using both artificial and real data video streams collected from monitoring devices.
25
KODAMA, KOICHI, KOHEI SUENAGA, and NAOKI KOBAYASHI. "Translation of tree-processing programs into stream-processing programs based on ordered linear type." Journal of Functional Programming 18, no. 3 (May 2008): 333–71. http://dx.doi.org/10.1017/s0956796807006570.
Full textAbstract:
AbstractThere are two ways to write a program for manipulating tree-structured data such as XML documents: One is to write a tree-processing program focusing on the logical structure of the data and the other is to write a stream-processing program focusing on the physical structure. While tree-processing programs are easier to write than stream-processing programs, tree-processing programs are less efficient in memory usage since they use trees as intermediate data. Our aim is to establish a method for automatically translating a tree-processing program to a stream-processing one in order to take the best of both worlds. We first define a programming language for processing binary trees and a type system based on ordered linear type, and show that every well-typed program can be translated to an equivalent stream-processing program. We then extend the language and the type system to deal with XML documents. We have implemented an XML stream processor generator based on our algorithm, and obtained promising experimental results.
26
Li, Guang Di, Guo Yin Wang, Xue Rui Zhang, Wei Hui Deng, and Fan Zhang. "Forest Cover Types Classification Based on Online Machine Learning on Distributed Cloud Computing Platforms of Storm and SAMOA." Advanced Materials Research 955-959 (June 2014): 3803–12. http://dx.doi.org/10.4028/www.scientific.net/amr.955-959.3803.
Full textAbstract:
Storm is the most popular realtime stream processing platform, which can be used to deal with online machine learning. Similar to how Hadoop provides a set of general primitives for doing batch processing, Storm provides a set of general primitives for doing realtime computation. SAMOA includes distributed algorithms for the most common machine learning tasks like Mahout for Hadoop. SAMOA is both a platform and a library. In this paper, Forest cover types, a large benchmaking dataset available at the UCI KDD Archive is used as the data stream source. Vertical Hoeffding Tree, a parallelizing streaming decision tree induction for distributed enviroment, which is incorporated in SAMOA API is applied on Storm platform. This study compared stream prcessing technique for predicting forest cover types from cartographic variables with traditional classic machine learning algorithms applied on this dataset. The test then train method used in this system is totally different from the traditional train then test. The results of the stream processing technique indicated that it’s output is aymptotically nearly identical to that of a conventional learner, but the model derived from this system is totally scalable, real-time, capable of dealing with evolving streams and insensitive to stream ordering.
27
Hassan, Alaa Abdelraheem, and Tarig Mohammed Hassan. "Real-Time Big Data Analytics for Data Stream Challenges: An Overview." European Journal of Information Technologies and Computer Science 2, no. 4 (July 25, 2022): 1–6. http://dx.doi.org/10.24018/compute.2022.2.4.62.
Full textAbstract:
The conventional approach of evaluating massive data is inappropriate for real-time analysis; therefore, analysing big data in a data stream remains a critical issue for numerous applications. It is critical in real-time big data analytics to process data at the point where they are arriving at a quick reaction and good decision making, necessitating the development of a novel architecture that allows for real-time processing at high speed and low latency. Processing and anlayzing a data stream in real-time is critical for a variety of applications; however, handling a large amount of data from a variety of sources, such as sensor networks, web traffic, social media, video streams, and other sources, is a considerable difficulty. The main goal of this paper is to give an overview of the current architecture for real time big data analytics, real-time data stream processing methods available, including their system architectures Lambda, kappa, and delta large data stream processing.
28
Jovanovic, Zeljko. "Data stream management system for moving sensor object data." Serbian Journal of Electrical Engineering 12, no. 1 (2015): 117–27. http://dx.doi.org/10.2298/sjee1501117j.
Full textAbstract:
Sensor and communication development has led to the development of new types of applications. Classic database data storage becomes inadequate when data streams arrive from multiple sensors. Then, data querying and result presentation are not efficient. The desired results are obtained with a delay, and the database is filled with a large amount of unnecessary data. To adequately support the above applications, Data Stream Management System (DSMS) applications are needed. DSMSs provide real-time data stream processing. In this paper, a client-server system is presented with DSMS realized on the Java WebDSMS application server side. WebDSMS functionalities are tested with simulated data and in real-life usage.
29
Wang, Yongheng, Xiaozan Zhang, and Zengwang Wang. "A Proactive Decision Support System for Online Event Streams." International Journal of Information Technology & Decision Making 17, no. 06 (November 2018): 1891–913. http://dx.doi.org/10.1142/s0219622018500463.
Full textAbstract:
In-stream big data processing is an important part of big data processing. Proactive decision support systems can predict future system states and execute some actions to avoid unwanted states. In this paper, we propose a proactive decision support system for online event streams. Based on Complex Event Processing (CEP) technology, this method uses structure varying dynamic Bayesian network to predict future events and system states. Different Bayesian network structures are learned and used according to different event context. A networked distributed Markov decision processes model with predicting states is proposed as sequential decision making model. A Q-learning method is investigated for this model to find optimal joint policy. The experimental evaluations show that this method works well for congestion control in transportation system.
30
McFerren, G., and T. van Zyl. "GEOSPATIAL DATA STREAM PROCESSING IN PYTHON USING FOSS4G COMPONENTS." ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLI-B7 (June 22, 2016): 931–37. http://dx.doi.org/10.5194/isprs-archives-xli-b7-931-2016.
Full textAbstract:
One viewpoint of current and future IT systems holds that there is an increase in the scale and velocity at which data are acquired and analysed from heterogeneous, dynamic sources. In the earth observation and geoinformatics domains, this process is driven by the increase in number and types of devices that report location and the proliferation of assorted sensors, from satellite constellations to oceanic buoy arrays. Much of these data will be encountered as self-contained messages on data streams - continuous, infinite flows of data. Spatial analytics over data streams concerns the search for spatial and spatio-temporal relationships within and amongst data “on the move”. In spatial databases, queries can assess a store of data to unpack spatial relationships; this is not the case on streams, where spatial relationships need to be established with the incomplete data available. Methods for spatially-based indexing, filtering, joining and transforming of streaming data need to be established and implemented in software components. This article describes the usage patterns and performance metrics of a number of well known FOSS4G Python software libraries within the data stream processing paradigm. In particular, we consider the RTree library for spatial indexing, the Shapely library for geometric processing and transformation and the PyProj library for projection and geodesic calculations over streams of geospatial data. We introduce a message oriented Python-based geospatial data streaming framework called Swordfish, which provides data stream processing primitives, functions, transports and a common data model for describing messages, based on the Open Geospatial Consortium Observations and Measurements (O&M) and Unidata Common Data Model (CDM) standards. We illustrate how the geospatial software components are integrated with the Swordfish framework. Furthermore, we describe the tight temporal constraints under which geospatial functionality can be invoked when processing high velocity, potentially infinite geospatial data streams. The article discusses the performance of these libraries under simulated streaming loads (size, complexity and volume of messages) and how they can be deployed and utilised with Swordfish under real load scenarios, illustrated by a set of Vessel Automatic Identification System (AIS) use cases. We conclude that the described software libraries are able to perform adequately under geospatial data stream processing scenarios - many real application use cases will be handled sufficiently by the software.
31
Peppin, William A., and Walter F. Nicks. "Real-Time Analog and Digital Data Acquisition Through CUSP." Seismological Research Letters 63, no. 2 (April 1, 1992): 181–89. http://dx.doi.org/10.1785/gssrl.63.2.181.
Full textAbstract:
Abstract The University of Nevada Seismological Laboratory operates an array of 60 analog short-period and 10 three-component digital telemetered seismic stations, 90 data traces in all, in Nevada and eastern California. Formerly, the seismic data streams were recorded and processed on three separate computers running disparate software and writing incompatible data formats which made access to the digital data quite cumbersome. These systems were recently replaced by a single computer system, a MicroVAX II running VAX/VMS, together with Generic CUSP (Caltech -U.S.G.S. Seismic Processing System), a controlled software system from the U.S.G.S. in Menlo Park. Telemetered digital data are stored simultaneously in two ways, unique to this network. First, these digital data are brought asynchronously into the computer using a standard direct-memory access interface and recorded continuously on an Exabyte 8-mm helical-scan tapedrive. Second, the digital data are passed through a D to A converter and intermixed with the incoming analog data stream used for routine network processing. This analog data stream is then itself digitized and presented to the computer. In this way, calibrated digital waveforms are available in the routine data processing stream, now entirely comprised of digital waveforms, used to locate earthquakes. At the same time, this allows easy access to these data in research applications involving the processing of seismic waveforms.
32
Umar, M., M. I. Ofem, A. S. Anwar, and M. M. Usman. "Electrical Conductivity of PA6/Graphite and Graphite Nanoplatelets Composites using Two Processing Streams." March 2021 5, no. 1 (March 2021): 19–31. http://dx.doi.org/10.36263/nijest.2021.01.0251.
Full textAbstract:
The percolation threshold (PT) of any polymer/particulate carbon composite depends on the processing, the dispersed state of the filler, the matrix used and the morphology attained. Sonication technique was used to make PA6/G and PA6/GNP composites employing in situ polymerisation, after which their electrical conductivity behaviours were investigated. While overhead stirring and horn sonication were used to distribute and disperse the carbon fillers, the composites were made in 2 streams 40/10 and 20/20. The 40/10 stream implies that while dispersing the carbon fillers in PA6 monomer, 40% amplitude of sonication was applied for 10 minutes whereas the 20/20 stream implies 20% amplitude of sonication for 20 minutes. In both streams, the dispersing strain imparted on the monomer/carbon mixture was 400 in magnitude. Purely ohmic electrical conductivity behaviour was attained at 9.75 G wt. % for IG 40/10 system. For composites in the IG 20/20 system, same was attained at 10.00 G wt. %. Electrical conductivity sufficient for electrostatic discharge applications was achieved above 15 G wt. % in the IG 40/10 system. Using the power law percolation theory, percolation threshold was attained at 9.7 G wt. % loading in IG 40/10 system, while same was attained at 7.6 G wt. % loading in IG 20/20 system. For the GNP based systems, percolation threshold occurred at 5.2 GNP wt. % in the INP 40/10 system whereas same occurred at 7.4 GNP wt. % in the IG 20/20 system.
33
Liu, Jun Qiang, and Xiao Ling Guan. "Composite Event Processing for Data Streams and Domain Knowledge." Advanced Materials Research 219-220 (March 2011): 927–31. http://dx.doi.org/10.4028/www.scientific.net/amr.219-220.927.
Full textAbstract:
In recent years the processing of composite event queries over data streams has attracted a lot of research attention. Traditional database techniques were not designed for stream processing system. Furthermore, example continuous queries are often formulated in declarative query language without specifying the semantics. To overcome these deficiencies, this article presents the design, implementation, and evaluation of a system that executes data streams with semantic information. Then, a set of optimization techniques are proposed for handling query. So, our approach not only makes it possible to express queries with a sound semantics, but also provides a solid foundation for query optimization. Experiment results show that our approach is effective and efficient for data streams and domain knowledge.
34
Cai, Walter, Philip A. Bernstein, Wentao Wu, and Badrish Chandramouli. "Optimization of threshold functions over streams." Proceedings of the VLDB Endowment 14, no. 6 (February 2021): 878–89. http://dx.doi.org/10.14778/3447689.3447693.
Full textAbstract:
A common stream processing application is alerting, where the data stream management system (DSMS) continuously evaluates a threshold function over incoming streams. If the threshold is crossed, the DSMS raises an alarm. The threshold function is often calculated over two or more streams, such as combining temperature and humidity readings to determine if moisture will form on a machine and therefore cause it to malfunction. This requires taking a temporal join across the input streams. We show that for the broad class of functions called quasiconvex functions, the DSMS needs to retain very few tuples per-data-stream for any given time interval and still never miss an alarm. This surprising result yields a large memory savings during normal operation. That savings is also important if one stream fails, since the DSMS would otherwise have to cache all tuples in other streams until the failed stream recovers. We prove our algorithm is optimal and provide experimental evidence that validates its substantial memory savings.
35
Zhai, Hong Yu, Li Li, and Hong Hua Xu. "The Design of Query Processing in Data Stream Management System." Advanced Materials Research 952 (May 2014): 351–54. http://dx.doi.org/10.4028/www.scientific.net/amr.952.351.
Full textAbstract:
data stream management system is used to manage and query coming large, continuous, fast and flexible data stream. The system is based on the flow of data extraction, transformation, combination, which is the main content and task query execution. This paper mainly discusses the design and implementation of query execution module and query execution is composed of two parts which include query operations, query execution and scheduling.
36
Chang Hyun Park, Hyung Rim Choi, Byung Kwon Park, and Young Jae Park. "A Continuous Query Processing System for RFID Data Stream." INTERNATIONAL JOURNAL ON Advances in Information Sciences and Service Sciences 5, no. 8 (April 30, 2013): 1282–89. http://dx.doi.org/10.4156/aiss.vol5.issue8.150.
Full text
37
Han, Seungchul, and Hyunchul Kang. "A Continuous Query Processing System for XML Stream Data." KIPS Transactions:PartD 11D, no. 7 (December 1, 2004): 1375–84. http://dx.doi.org/10.3745/kipstd.2004.11d.7.1375.
Full text
38
Balazinska, Magdalena, Hari Balakrishnan, Samuel R. Madden, and Michael Stonebraker. "Fault-tolerance in the borealis distributed stream processing system." ACM Transactions on Database Systems 33, no. 1 (March 2008): 1–44. http://dx.doi.org/10.1145/1331904.1331907.
Full text
39
Hu, Liang, Rui Sun, Feng Wang, Xiuhong Fei, and Kuo Zhao. "A Stream Processing System for Multisource Heterogeneous Sensor Data." Journal of Sensors 2016 (2016): 1–8. http://dx.doi.org/10.1155/2016/4287834.
Full textAbstract:
With the rapid development of the Internet of Things (IoT), a variety of sensor data are generated around everyone’s life. New research perspective regarding the streaming sensor data processing of the IoT has been raised as a hot research topic that is precisely the theme of this paper. Our study serves to provide guidance regarding the practical aspects of the IoT. Such guidance is rarely mentioned in the current research in which the focus has been more on theory and less on issues describing how to set up a practical system. In our study, we employ numerous open source projects to establish a distributed real time system to process streaming data of the IoT. Two urgent issues have been solved in our study that are (1) multisource heterogeneous sensor data integration and (2) processing streaming sensor data in real time manner with low latency. Furthermore, we set up a real time system to process streaming heterogeneous sensor data from multiple sources with low latency. Our tests are performed using field test data derived from environmental monitoring sensor data collected from indoor environment for system validation. The results show that our proposed system is valid and efficient for multisource heterogeneous sensor data integration and streaming data processing in real time manner.
40
Böhme, C., P. Bouwer, and M. J. Prinsloo. "Real-time stream processing for active fire monitoring on Landsat 8 direct reception data." ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XL-7/W3 (April 29, 2015): 765–70. http://dx.doi.org/10.5194/isprsarchives-xl-7-w3-765-2015.
Full textAbstract:
Some remote sensing applications are relatively time insensitive, for others, near-real-time processing (results 30-180 minutes after data reception) offer a viable solution. There are, however, a few applications, such as active wildfire monitoring or ship and airplane detection, where real-time processing and image interpretation offers a distinct advantage. The objective of real-time processing is to provide notifications before the complete satellite pass has been received. This paper presents an automated system for real-time, stream–based processing of data acquired from direct broadcast push-broom sensors for applications that require a high degree of timeliness. Based on this system, a processing chain for active fire monitoring using Landsat 8 live data streams was implemented and evaluated. The real-time processing system, called the FarEarth Observer, is connected to a ground station’s demodulator and uses its live data stream as input. Processing is done on variable size image segments assembled from detector lines of the push broom sensor as they are streamed from the satellite, enabling detection of active fires and sending of notifications within seconds of the satellite passing over the affected area, long before the actual acquisition completes. This approach requires performance optimized techniques for radiometric and geometric correction of the sensor data. Throughput of the processing system is kept well above the 400Mbit/s downlink speed of Landsat 8. A latency of below 10 seconds from sensor line acquisition to anomaly detection and notification is achieved. Analyses of geometric and radiometric accuracy and comparisons in latency to traditional near-real-time systems are also presented.
41
Okoro, Oseweuba, Zhifa Sun, and John Birch. "Techno-Economic Assessment of a Scaled-Up Meat Waste Biorefinery System: A Simulation Study." Materials 12, no. 7 (March 28, 2019): 1030. http://dx.doi.org/10.3390/ma12071030.
Full textAbstract:
While exports from the meat industry in New Zealand constitute a valuable source of foreign exchange, the meat industry is also responsible for the generation of large masses of waste streams. These meat processing waste streams are largely biologically unstable and are capable of leading to unfavourable environmental outcomes if not properly managed. To enable the effective management of the meat processing waste streams, a value-recovery based strategy, for the complete valorisation of the meat processing waste biomass, is proposed. In the present study therefore, a biorefinery system that integrates the biomass conversion technologies of hydrolysis, esterification, anaerobic digestion and hydrothermal liquefaction has been modelled, simulated and optimized for enhanced environmental performance and economic performance. It was determined that an initial positive correlation between the mass feed rate of the waste to the biorefinery system and its environmental performance exists. However, beyond an optimal total mass feed rate of the waste stream there is a deterioration of the environmental performance of the biorefinery system. It was also determined that economies of scale ensure that any improvement in the economic performance of the biorefinery system with increasing total mass feed rate of the waste stream, is sustained. The present study established that the optimized meat waste biorefinery system facilitated a reduction in the unit production costs of the value-added products of biodiesel, biochar and biocrude compared the literature-obtained unit production costs of the respective aforementioned products when generated from stand-alone systems. The unit production cost of biogas was however shown to be comparable to the literature-obtained unit production cost of biogas. Finally, the present study showed that the optimized meat processing waste biorefinery could achieve enhanced economic performance while simultaneously maintaining favourable environmental sustainability.
42
Fais, Alessandra, Giuseppe Lettieri, Gregorio Procissi, Stefano Giordano, and Francesco Oppedisano. "Data Stream Processing for Packet-Level Analytics." Sensors 21, no. 5 (March 3, 2021): 1735. http://dx.doi.org/10.3390/s21051735.
Full textAbstract:
One of the most challenging tasks for network operators is implementing accurate per-packet monitoring, looking for signs of performance degradation, security threats, and so on. Upon critical event detection, corrective actions must be taken to keep the network running smoothly. Implementing this mechanism requires the analysis of packet streams in a real-time (or close to) fashion. In a softwarized network context, Stream Processing Systems (SPSs) can be adopted for this purpose. Recent solutions based on traditional SPSs, such as Storm and Flink, can support the definition of general complex queries, but they show poor performance at scale. To handle input data rates in the order of gigabits per seconds, programmable switch platforms are typically used, although they offer limited expressiveness. With the proposed approach, we intend to offer high performance and expressive power in a unified framework by solely relying on SPSs for multicores. Captured packets are translated into a proper tuple format, and network monitoring queries are applied to tuple streams. Packet analysis tasks are expressed as streaming pipelines, running on general-purpose programmable network devices, and a second stage of elaboration can process aggregated statistics from different devices. Experiments carried out with an example monitoring application show that the system is able to handle realistic traffic at a 10 Gb/s speed. The same application scales almost up to 20 Gb/s speed thanks to the simple optimizations of the underlying framework. Hence, the approach proves to be viable and calls for the investigation of more extensive optimizations to support more complex elaborations and higher data rates.
43
Njemanze, Philip, Mathias Kranz, and Peter Brust. "Fourier Analysis of Cerebral Metabolism of Glucose: Gender Differences in Mechanisms of Color Processing in the Ventral and Dorsal Streams in Mice." Forecasting 1, no. 1 (September 30, 2018): 135–56. http://dx.doi.org/10.3390/forecast1010010.
Full textAbstract:
Conventional imaging methods could not distinguish processes within the ventral and dorsal streams. The application of Fourier time series analysis was helpful to segregate changes in the ventral and dorsal streams of the visual system in male and female mice. The present study measured the accumulation of [18F]fluorodeoxyglucose ([18F]FDG) in the mouse brain using small animal positron emission tomography and magnetic resonance imaging (PET/MRI) during light stimulation with blue and yellow filters, compared to during conditions of darkness. Fourier analysis was performed using mean standardized uptake values (SUV) of [18F]FDG for each stimulus condition to derive spectral density estimates for each condition. In male mice, luminance opponency occurred by S-peak changes in the sub-cortical retino-geniculate pathways in the dorsal stream supplied by ganglionic arteries in the left visual cortex, while chromatic opponency involved C-peak changes in the cortico-subcortical pathways in the ventral stream perfused by cortical arteries in the left visual cortex. In female mice, there was resonance phenomenon at C-peak in the ventral stream perfused by the cortical arteries in the right visual cortex during luminance processing. Conversely, chromatic opponency caused by S-peak changes in the subcortical retino-geniculate pathways in the dorsal stream supplied by the ganglionic arteries in the right visual cortex. In conclusion, Fourier time series analysis uncovered distinct mechanisms of color processing in the ventral stream in males, while in female mice color processing was in the dorsal stream. It demonstrated that computation of colour processing as a conscious experience could have a wide range of applications in neuroscience, artificial intelligence and quantum mechanics.
44
Liu, Haitao, Qingkui Chen, and Puchen Liu. "An Optimization Method of Large-Scale Video Stream Concurrent Transmission for Edge Computing." Mathematics 11, no. 12 (June 8, 2023): 2622. http://dx.doi.org/10.3390/math11122622.
Full textAbstract:
Concurrent access to large-scale video data streams in edge computing is an important application scenario that currently faces a high cost of network access equipment and high data packet loss rate. To solve this problem, a low-cost link aggregation video stream data concurrent transmission method is proposed. Data Plane Development Kit (DPDK) technology supports the concurrent receiving and forwarding function of multiple Network Interface Cards (NICs). The Q-learning data stream scheduling model is proposed to solve the load scheduling of multiple queues of multiple NICs. The Central Processing Unit (CPU) transmission processing unit was dynamically selected by data stream classification, as well as a reward function, to achieve the dynamic load balancing of data stream transmission. The experiments conducted demonstrate that this method expands the bandwidth by 3.6 times over the benchmark scheme for a single network port, and reduces the average CPU load ratio by 18%. Compared to the UDP and DPDK schemes, it lowers the average system latency by 21%, reduces the data transmission packet loss rate by 0.48%, and improves the overall system transmission throughput. This transmission optimization scheme can be applied in data centers and edge computing clusters to improve the communication performance of big data processing.
45
Bhatt, Nirav, and Amit Thakkar. "An efficient approach for low latency processing in stream data." PeerJ Computer Science 7 (March 10, 2021): e426. http://dx.doi.org/10.7717/peerj-cs.426.
Full textAbstract:
Stream data is the data that is generated continuously from the different data sources and ideally defined as the data that has no discrete beginning or end. Processing the stream data is a part of big data analytics that aims at querying the continuously arriving data and extracting meaningful information from the stream. Although earlier processing of such stream was using batch analytics, nowadays there are applications like the stock market, patient monitoring, and traffic analysis which can cause a drastic difference in processing, if the output is generated in levels of hours and minutes. The primary goal of any real-time stream processing system is to process the stream data as soon as it arrives. Correspondingly, analytics of the stream data also needs consideration of surrounding dependent data. For example, stock market analytics results are often useless if we do not consider their associated or dependent parameters which affect the result. In a real-world application, these dependent stream data usually arrive from the distributed environment. Hence, the stream processing system has to be designed, which can deal with the delay in the arrival of such data from distributed sources. We have designed the stream processing model which can deal with all the possible latency and provide an end-to-end low latency system. We have performed the stock market prediction by considering affecting parameters, such as USD, OIL Price, and Gold Price with an equal arrival rate. We have calculated the Normalized Root Mean Square Error (NRMSE) which simplifies the comparison among models with different scales. A comparative analysis of the experiment presented in the report shows a significant improvement in the result when considering the affecting parameters. In this work, we have used the statistical approach to forecast the probability of possible data latency arrives from distributed sources. Moreover, we have performed preprocessing of stream data to ensure at-least-once delivery semantics. In the direction towards providing low latency in processing, we have also implemented exactly-once processing semantics. Extensive experiments have been performed with varying sizes of the window and data arrival rate. We have concluded that system latency can be reduced when the window size is equal to the data arrival rate.
46
Dr. Pasumponpandian. "Analysis of Data Stream Processing At Edge Layer for Internet of Things." Journal of ISMAC 2, no. 1 (March 15, 2020): 26–37. http://dx.doi.org/10.36548/jismac.2020.1.003.
Full textAbstract:
The progress of internet of things at a rapid pace and simultaneous development of the technologies and the processing capabilities has paved way for the development of decentralized systems that are relying on cloud services. Though the decentralized systems are founded on cloud complexities still prevail in transferring all the information’s that are been sensed through the IOT devices to the cloud. This because of the huge streams of information’s gathered by certain applications and the expectation to have a timely response, incurring minimized delay, computing energy and enhanced reliability. So this kind of decentralization has led to the development of middle layer between the cloud and the IOT, and was termed as the Edge layer, meaning bringing down the service of the cloud to the user edge. The paper puts forth the analysis of the data stream processing in the edge layer taking in the complexities involved in the computing the data streams of IOT in an edge layer and puts forth the real time analytics in the edge layer to examine the data streams of the internet of things offering a data- driven insight for parking system in the smart cities.
47
Poźniak, Krzysztof. "Modeling of Synchronous Data Streams Processing in the RPC Muon Trigger System of the CMS Experiment." International Journal of Electronics and Telecommunications 56, no. 4 (November 1, 2010): 489–502. http://dx.doi.org/10.2478/v10177-010-0067-3.
Full textAbstract:
Modeling of Synchronous Data Streams Processing in the RPC Muon Trigger System of the CMS ExperimentThis paper presents signal synchronization aspects in a large, distributed, multichannel RPC Muon Trigger system in the CMS experiment. The paper is an introduction to normalized structure analysis methods of such systems. The method introduces a general model of the system, presented in a form of a network of distributed, synchronous, pipeline processes. The model is based on a definition of a synchronous data stream and its formal, fundamental properties. Theoretical considerations are supported by a practical application of synchronous streams and processes management. The following processes were modeled and implemented in hardware: window synchronization, derandomization, data concentration and generation of test pulses. There are presented chosen results of the model application in the CMS experiment.
48
Zou, Yong Gui, Yuan Lei Tang, and Ying Xia. "Load Balancing Algorithm of Stream Data Based on Correlation Analysis." Applied Mechanics and Materials 543-547 (March 2014): 2594–99. http://dx.doi.org/10.4028/www.scientific.net/amm.543-547.2594.
Full textAbstract:
With the development of stream processing technology, achieving load balance of resource accessing becomes one of key problems. However, the existing technologies are difficult to balance the system and maintain the data integrity requirements when large data stream arrive. In this paper, combined with the dynamic load balancing algorithm, we propose a load balancing algorithm of stream data based on correlation analysis (SDCA-LBA). The algorithm analyses correlation of stream data through the window feature statistics. On the basis of ensuring the load balancing of the stream data system, we consider the data correlation. The experiment results show that this method can effectively solve the problem of stream data load balancing, maintain the data integrity, and improve the system data processing capabilities.
49
Ni, Xiang, Jing Li, Mo Yu, Wang Zhou, and Kun-Lung Wu. "Generalizable Resource Allocation in Stream Processing via Deep Reinforcement Learning." Proceedings of the AAAI Conference on Artificial Intelligence 34, no. 01 (April 3, 2020): 857–64. http://dx.doi.org/10.1609/aaai.v34i01.5431.
Full textAbstract:
This paper considers the problem of resource allocation in stream processing, where continuous data flows must be processed in real time in a large distributed system. To maximize system throughput, the resource allocation strategy that partitions the computation tasks of a stream processing graph onto computing devices must simultaneously balance workload distribution and minimize communication. Since this problem of graph partitioning is known to be NP-complete yet crucial to practical streaming systems, many heuristic-based algorithms have been developed to find reasonably good solutions. In this paper, we present a graph-aware encoder-decoder framework to learn a generalizable resource allocation strategy that can properly distribute computation tasks of stream processing graphs unobserved from training data. We, for the first time, propose to leverage graph embedding to learn the structural information of the stream processing graphs. Jointly trained with the graph-aware decoder using deep reinforcement learning, our approach can effectively find optimized solutions for unseen graphs. Our experiments show that the proposed model outperforms both METIS, a state-of-the-art graph partitioning algorithm, and an LSTM-based encoder-decoder model, in about 70% of the test cases.
50
Fu, Gang, Ming Xin Kou, and Ren Long Li. "Design and Implementation of Driving Mechanism on Software Aerospace Measurement and Control System." Advanced Materials Research 989-994 (July 2014): 3084–87. http://dx.doi.org/10.4028/www.scientific.net/amr.989-994.3084.
Full textAbstract:
According to the signal processing unit in aerospace measurement and control system between the flow of water features, this paper proposes a software suitable for aerospace measurement and control system of driving mechanism. This paper first introduces the basic structure of aerospace measurement and control system software, having studied the static and dynamic data stream driving mechanism on the basis of detailed discusses the design and implementation process of this kind of driving mechanism of data stream. It adopts the method of message control, according to the dynamic data flow driven mechanism, realize the process of the signal processing unit and each signal processing unit between the data flow between the internal thread. Compared the same sort of dynamic data stream driving mechanism, the drive mechanism possesses the advantages of flexibility and easy to implement.