Academic literature on the topic 'Stream Processing System'
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the lists of relevant articles, books, theses, conference reports, and other scholarly sources on the topic 'Stream Processing System.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Journal articles on the topic "Stream Processing System"
1
Shuiying Yu, Shuiying Yu, Yinting Zheng Shuiying Yu, Fan Zhang Yinting Zheng, Hanhua Chen Fan Zhang, and Hai Jin Hanhua Chen. "TriJoin: A Time-Efficient and Scalable Three-Way Distributed Stream Join System." 網際網路技術學刊 24, no. 2 (March 2023): 475–85. http://dx.doi.org/10.53106/160792642023032402024.
Full textAbstract:
<p>Stream join is one of the most fundamental operations in data stream processing applications. Existing distributed stream join systems can support efficient two-way join, which is a join operation between two streams. Based the two-way join, implementing a three-way join require to be split into double two-way joins, where the second two-way join needs to wait for the join result transmitted from the first two-way join. We show through experiments that such a design raises prohibitively high processing latency. To solve this problem, we propose TriJoin, a time-efficient three-way distributed stream join system. We design a symmetric wait-free structure by symmetrically partitioning tuples and reused join. TriJoin utilizes reused join to join each new tuple with the intermediate result of the other two streams and stored tuples locally. For a new tuple, TriJoin only joins it with the intermediate result to generate the final result without waiting, greatly reducing the processing latency. In TriJoin, we design two partitioning and storage schemes according to two different forms of three-way stream join. We implement TriJoin and conduct comprehensive experiments to evaluate the performance using real-world traces. Results show that TriJoin significantly reduces the processing latency by up to 68%, compared to existing designs.</p> <p> </p>
2
Shi, Peng, and Li Li. "Design of Network Analysis System Based on Stream Computing." Journal of Computational and Theoretical Nanoscience 14, no. 1 (January 1, 2017): 64–68. http://dx.doi.org/10.1166/jctn.2017.6125.
Full textAbstract:
The functions of the network analysis system include detection and analysis of network data stream. According to the results of the network analysis, we monitor the network accident and avoid the security risks. This can improve the network performance and increase the network availability. As the data flow in the network is constantly produced, the biggest characteristic of network analysis system is that it is a real-time system. Because of the high requirements of the network data analysis and network fault processing, the system requires very high processing efficiency of the real time data of network. Stream computing is a technique specifically for processing real-time data streams. Its idea is that the value of the data is reduced with the lapse of time, so as long as the data appearing, it must be processed as soon as possible. So we use the technology of stream computing to design network analysis system to meet the needs of real-time capability. Moreover, the stream computing framework has been widely welcomed in the field because of its good expansibility, ease of use and flexibility. In this paper, firstly, we introduce the characteristics of the data processing based on stream computing and the traditional data processing separately. We point out their difference and introduce the technique of stream computing. Then, we introduce the architecture of network analysis system designed base on the technique of stream computing. The architecture includes two main components that are logic processing layer and communication layer. We describe the characteristics of each component and functional characteristics in detail, and we introduce the system load balancing algorithm. Finally, by experiments, we verify the effectiveness of the system’s characteristics of dynamic expansion and load balancing.
3
Bernardelli de Moraes, Matheus, and André Leon Sampaio Gradvohl. "Evaluating the impact of a coordinated checkpointing in distributed data streams processing systems using discrete event simulation." Revista Brasileira de Computação Aplicada 12, no. 2 (May 19, 2020): 16–27. http://dx.doi.org/10.5335/rbca.v12i2.10295.
Full textAbstract:
Data Streams Processing systems process continuous flows of data under Quality of Service requirements. Data streams often contain critical information which requires real-time processing. To guarantee systems' dependability and avoid information loss, one must use a fault-tolerance strategy. However, there are several strategies available, and the proper evaluation of which mechanism is better for each system architecture is challenging, especially in large-scale distributed systems. In this paper, we propose a discrete simulation model for investigating the impacts of the Coordinated Checkpoint fault tolerance strategy imposes on Data Stream Processing Systems. Results show that this strategy critically affects stream processing in failure-prone situations due to an increase in latency up to 120% and information loss, reaching 95% of the processing window in the worst case.
4
Valeev, S. S., N. V. Kondratyeva, A. S. Kovtunenko, M. A. Timirov, and R. R. Karimov. "Distributed stream data processing system in multi-agent safety system of infrastructure objects." Information Technology and Nanotechnology, no. 2416 (2019): 324–31. http://dx.doi.org/10.18287/1613-0073-2019-2416-324-331.
Full textAbstract:
The solution of the problem of resource management in distributed computing systems of processing stream data in safety systems of distributed objects is considered. The tasks of streaming data processing in a multi-level multi-agent evacuation system in an infrastructure object are considered. The features of the mathematical model of a distributed stream data processing system are discussed.
5
Ye, Qian, and Minyan Lu. "s2p: Provenance Research for Stream Processing System." Applied Sciences 11, no. 12 (June 15, 2021): 5523. http://dx.doi.org/10.3390/app11125523.
Full textAbstract:
The main purpose of our provenance research for DSP (distributed stream processing) systems is to analyze abnormal results. Provenance for these systems is not nontrivial because of the ephemerality of stream data and instant data processing mode in modern DSP systems. Challenges include but are not limited to an optimization solution for avoiding excessive runtime overhead, reducing provenance-related data storage, and providing it in an easy-to-use fashion. Without any prior knowledge about which kinds of data may finally lead to the abnormal, we have to track all transformations in detail, which potentially causes hard system burden. This paper proposes s2p (Stream Process Provenance), which mainly consists of online provenance and offline provenance, to provide fine- and coarse-grained provenance in different precision. We base our design of s2p on the fact that, for a mature online DSP system, the abnormal results are rare, and the results that require a detailed analysis are even rarer. We also consider state transition in our provenance explanation. We implement s2p on Apache Flink named as s2p-flink and conduct three experiments to evaluate its scalability, efficiency, and overhead from end-to-end cost, throughput, and space overhead. Our evaluation shows that s2p-flink incurs a 13% to 32% cost overhead, 11% to 24% decline in throughput, and few additional space costs in the online provenance phase. Experiments also demonstrates the s2p-flink can scale well. A case study is presented to demonstrate the feasibility of the whole s2p solution.
6
Al Jawarneh, Isam Mashhour, Paolo Bellavista, Antonio Corradi, Luca Foschini, and Rebecca Montanari. "QoS-Aware Approximate Query Processing for Smart Cities Spatial Data Streams." Sensors 21, no. 12 (June 17, 2021): 4160. http://dx.doi.org/10.3390/s21124160.
Full textAbstract:
Large amounts of georeferenced data streams arrive daily to stream processing systems. This is attributable to the overabundance of affordable IoT devices. In addition, interested practitioners desire to exploit Internet of Things (IoT) data streams for strategic decision-making purposes. However, mobility data are highly skewed and their arrival rates fluctuate. This nature poses an extra challenge on data stream processing systems, which are required in order to achieve pre-specified latency and accuracy goals. In this paper, we propose ApproxSSPS, which is a system for approximate processing of geo-referenced mobility data, at scale with quality of service guarantees. We focus on stateful aggregations (e.g., means, counts) and top-N queries. ApproxSSPS features a controller that interactively learns the latency statistics and calculates proper sampling rates to meet latency or/and accuracy targets. An overarching trait of ApproxSSPS is its ability to strike a plausible balance between latency and accuracy targets. We evaluate ApproxSSPS on Apache Spark Structured Streaming with real mobility data. We also compared ApproxSSPS against a state-of-the-art online adaptive processing system. Our extensive experiments prove that ApproxSSPS can fulfill latency and accuracy targets with varying sets of parameter configurations and load intensities (i.e., transient peaks in data loads versus slow arriving streams). Moreover, our results show that ApproxSSPS outperforms the baseline counterpart by significant magnitudes. In short, ApproxSSPS is a novel spatial data stream processing system that can deliver real accurate results in a timely manner, by dynamically specifying the limits on data samples.
7
Li, Huiyong, Xiaofeng Wu, and Yanhong Wang. "Dynamic Performance Analysis of STEP System in Internet of Vehicles Based on Queuing Theory." Computational Intelligence and Neuroscience 2022 (April 10, 2022): 1–13. http://dx.doi.org/10.1155/2022/8322029.
Full textAbstract:
The Internet of vehicles (IoV) is an important research area of the intelligent transportation systems using Internet of things theory. The complex event processing technology is a basic issue for processing the data stream in IoV. In recent years, many researchers process the temporal and spatial data flow by complex event processing technology. Spatial Temporal Event Processing (STEP) is a complex event query language focusing on the temporal and spatial data flow in Internet of vehicles. There are four processing models of the event stream processing system based on the complex event query language: finite automata model, matching tree model, directed acyclic graph model, and Petri net model. In addition, the worst-case response time of the event stream processing system is an important indicator of evaluating the performance of the system. Firstly, this paper proposed a core algorithm of the temporal and spatial event stream processing program based on STEP by Petri net model. Secondly, we proposed a novel method to estimate the worst-case response time of the event stream processing system, which is based on stochastic Petri net and queuing theory. Finally, through the simulation experiment based on queuing theory, this paper proves that the data stream processing system based on STEP has good dynamic performance in processing the spatiotemporal data stream in Internet of vehicles.
8
Akanbi, Adeyinka, and Muthoni Masinde. "A Distributed Stream Processing Middleware Framework for Real-Time Analysis of Heterogeneous Data on Big Data Platform: Case of Environmental Monitoring." Sensors 20, no. 11 (June 3, 2020): 3166. http://dx.doi.org/10.3390/s20113166.
Full textAbstract:
In recent years, the application and wide adoption of Internet of Things (IoT)-based technologies have increased the proliferation of monitoring systems, which has consequently exponentially increased the amounts of heterogeneous data generated. Processing and analysing the massive amount of data produced is cumbersome and gradually moving from classical ‘batch’ processing—extract, transform, load (ETL) technique to real-time processing. For instance, in environmental monitoring and management domain, time-series data and historical dataset are crucial for prediction models. However, the environmental monitoring domain still utilises legacy systems, which complicates the real-time analysis of the essential data, integration with big data platforms and reliance on batch processing. Herein, as a solution, a distributed stream processing middleware framework for real-time analysis of heterogeneous environmental monitoring and management data is presented and tested on a cluster using open source technologies in a big data environment. The system ingests datasets from legacy systems and sensor data from heterogeneous automated weather systems irrespective of the data types to Apache Kafka topics using Kafka Connect APIs for processing by the Kafka streaming processing engine. The stream processing engine executes the predictive numerical models and algorithms represented in event processing (EP) languages for real-time analysis of the data streams. To prove the feasibility of the proposed framework, we implemented the system using a case study scenario of drought prediction and forecasting based on the Effective Drought Index (EDI) model. Firstly, we transform the predictive model into a form that could be executed by the streaming engine for real-time computing. Secondly, the model is applied to the ingested data streams and datasets to predict drought through persistent querying of the infinite streams to detect anomalies. As a conclusion of this study, a performance evaluation of the distributed stream processing middleware infrastructure is calculated to determine the real-time effectiveness of the framework.
9
Otten, Lambert. "Wetdry composting of organic municipal solid waste: current status in Canada." Canadian Journal of Civil Engineering 28, S1 (January 1, 2001): 124–30. http://dx.doi.org/10.1139/l00-072.
Full textAbstract:
Source separation of municipal solid waste into wet and dry streams is proving to be an attractive alternative in dealing with solid waste, and in achieving provincial and national waste diversion objectives. The system provides important flexibility in the number of waste streams, collection methods, collection frequency, and waste processing. In the past few years, experience has been obtained with two-, three-, and four-stream source separation and collection, composting of the organic waste fraction, and recycling of the valuable dry waste. The systems used in Guelph, Ontario, Lunenburg, Nova Scotia, and Caledon, Ontario, are presented. Public interest and participation has been high, especially when a two-stream, mandatory system is used. Thus, the City of Guelph has reported a 98% participation rate in its two-stream system which means that the public accepted the two-stream approach. Experience has shown that, as the number of streams increase, there is a greater chance of putting waste in the wrong stream. There is a strong demand for compost at a bulk price of about $30/ t FOB at the plant. The processing cost of the three plants varied from $50/t to $80/t of waste received without allowing for credits derived from extended landfill life or reduction in environmental impact.Key words: municipal solid waste, organic, source-separation, composting.
10
Park, Alfred J., Cheng-Hong Li, Ravi Nair, Nobuyuki Ohba, Uzi Shvadron, Ayal Zaks, and Eugen Schenfeld. "Towards flexible exascale stream processing system simulation." SIMULATION 88, no. 7 (August 9, 2011): 832–51. http://dx.doi.org/10.1177/0037549711412981.
Full textDissertations / Theses on the topic "Stream Processing System"
1
Wladdimiro, Cottet Daniel. "Dynamic adaptation in Stream Processing Systems." Electronic Thesis or Diss., Sorbonne université, 2024. http://www.theses.fr/2024SORUS028.
Full textAbstract:
Le nombre de données produites par les systèmes ou applications Web actuels augmente rapidement en raison des nombreuses interactions avec les utilisateurs (dans le cadre par exemple, transactions boursières en temps réel, des jeux multijoueurs, des données en continu produits par Twitter, etc.). Ainsi, il existe une demande croissante, notamment dans les domaines du commerce, de la sécurité et de la recherche, pour des systèmes capables de traiter ces données en temps réel et de fournir des informations utiles dans un court laps de temps. Les systèmes de traitement des flux (SPS) répondent à ces besoins et ont été largement utilisés à cette fin. L’objectif des SPS est de traiter de grands volumes de données en temps réel en endentent un ensemble d’opérateurs dans des applications structurée en DAG. Le plupart des SPS existants, tels que Flink ou Storm, sont configurés avant leur déploiement, définissant généralement à l’avance le DAG et le nombre de répliques opérateurs. Une surestimation du nombre de répliques entraîne alors un gaspillage des ressources allouées. D’autre part, en fonction de l’interaction avec l’environnement, le taux de données en entrée peut fluctuer de manière dynamique et, par conséquent, les opérateurs peuvent être surchargés, ce qui entraîne une dégradation des performances du système. Ces SPS ne sont pas capables de s’adapter dynamiquement à la charge de travail de l’opérateur et aux variations du taux d’entrée. Pour résoudre ce problème, une solution consiste à augmenter dynamiquement le nombre de ressources, physiques ou logiques, allouées au SPS lorsque la demande de traitement d’un ou plusieurs opérateurs augmente. Nous présentons dans cette thèse deux approches, RA-SPS et PA-SPS, pour modifier dynamiquement le nombre de répliques d’un opérateur. L’approche réactive repose sur l’état courant des opérateurs calculé sur de multiples métriques. Tandis que le modèle prédictif se base sur la variation du taux d’entrée, le temps d’exécution des opérateurs et les événements en file d’attente. Nous avons également étendu Storm pour reconfigurer dynamiquement le nombre de copies sans avoir à geler l’application. Notre SPS met aussi en œuvre un équilibreur de charge qui distribue les événements entrants de manière équitable entre les répliques d’un opérateur. Des expériences sur la Google Cloud Platform (GCP) ont été menées avec des applications qui traitent le flux Twitter, le trafic DNS ou les traces de flux du journal système. Nous avons évalué différentes configurations et les avons comparées avec l’implémentation originale de Storm ainsi qu’avec des travaux de pointe tels que SPS DABS-Storm qui adapte également le nombre de répliques. Les résultat montrent que notre approche permet d’améliorer de manière conséquente le nombre d’événement traité tout en réduisant les coûtsThe amount of data produced by today’s web-based systems and applications increases rapidly, due to the many interactions with users (e.g. real-time stock market transactions, multiplayer games, streaming data produced by Twitter, etc.). As a result, there is a growing demand, particularly in the fields of commerce, security and research, for systems capable of processing this data in real time and providing useful information in a short space of time. Stream processing systems (SPS) meet these needs and have been widely used for this purpose. The aim of SPSs is to process large volumes of data in real time by housing a set of operators in applications based on Directed acyclic graphs (DAG). Most existing SPSs, such as Flink or Storm, are configured prior to deployment, usually defining the DAG and the number of operator replicas in advance. Overestimating the number of replicas can lead to a waste of allocated resources. On the other hand, depending on interaction with the environment, the rate of input data can fluctuate dynamically and, as a result, operators can become overloaded, leading to a degradation in system performance. These SPSs are not capable of dynamically adapting to operator workload and input rate variations. One solution to this problem is to dynamically increase the number of resources, physical or logical, allocated to the SPS when the processing demand of one or more operators increases. This thesis presents two SPSs, RA-SPS and PA-SPS, reactive and predictive approach respectively, for dynamically modifying the number of operator replicas. The reactive approach relies on the current state of operators computed on multiple metrics, while the predictive model is based on input rate variation, operator execution time, and queued events. The two SPSs extend Storm SPS to dynamically reconfigure the number of copies without having to downtime the application. They also implement a load balancer that distributes incoming events fairly among operator replicas. Experiments on the Google Cloud Platform (GCP) were carried out with applications that process Twitter data, DNS traffic, or logs traces. Performance was evaluated with different configurations and the results were compared with those of running the same applications on the original Storm as well as with state-of-the-art work such as SPS DABS-Storm, which also adapt the number of replicas. The comparison shows that both RA-SPS and PA-SPS can significantly improve the number of events processed, while reducing costs
2
Hongslo, Anders. "Stream Processing in the Robot Operating System framework." Thesis, Linköpings universitet, Artificiell intelligens och integrerad datorsystem, 2012. http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-79846.
Full textAbstract:
Streams of information rather than static databases are becoming increasingly important with the rapid changes involved in a number of fields such as finance, social media and robotics. DyKnow is a stream-based knowledge processing middleware which has been used in autonomous Unmanned Aerial Vehicle (UAV) research. ROS (Robot Operating System) is an open-source robotics framework providing hardware abstraction, device drivers, communication infrastructure, tools, libraries as well as other functionalities. This thesis describes a design and a realization of stream processing in ROS based on the stream-based knowledge processing middleware DyKnow. It describes how relevant information in ROS can be selected, labeled, merged and synchronized to provide streams of states. There are a lot of applications for such stream processing such as execution monitoring or evaluating metric temporal logic formulas through progression over state sequences containing the features of the formulas. Overviews are given of DyKnow and ROS before comparing the two and describing the design. The stream processing capabilities implemented in ROS are demonstrated through performance evaluations which show that such stream processing is fast and efficient. The resulting realization in ROS is also readily extensible to provide further stream processing functionality.
3
Kakkad, Vasvi. "Curracurrong: a stream processing system for distributed environments." Thesis, The University of Sydney, 2014. http://hdl.handle.net/2123/12861.
Full textAbstract:
Advances in technology have given rise to applications that are deployed on wireless sensor networks (WSNs), the cloud, and the Internet of things. There are many emerging applications, some of which include sensor-based monitoring, web traffic processing, and network monitoring. These applications collect large amount of data as an unbounded sequence of events and process them to generate a new sequences of events. Such applications need an adequate programming model that can process large amount of data with minimal latency; for this purpose, stream programming, among other paradigms, is ideal. However, stream programming needs to be adapted to meet the challenges inherent in running it in distributed environments. These challenges include the need for modern domain specific language (DSL), the placement of computations in the network to minimise energy costs, and timeliness in real-time applications. To overcome these challenges we developed a stream programming model that achieves easy-to-use programming interface, energy-efficient actor placement, and timeliness. This thesis presents Curracurrong, a stream data processing system for distributed environments. In Curracurrong, a query is represented as a stream graph of stream operators and communication channels. Curracurrong provides an extensible stream operator library and adapts to a wide range of applications. It uses an energy-efficient placement algorithm that optimises communication and computation. We extend the placement problem to support dynamically changing networks, and develop a dynamic program with polynomially bounded runtime to solve the placement problem. In many stream-based applications, real-time data processing is essential. We propose an approach that measures time delays in stream query processing; this model measures the total computational time from input to output of a query, i.e., end-to-end delay.
4
Tokmouline, Timur. "A signal oriented stream processing system for pipeline monitoring." Thesis, Massachusetts Institute of Technology, 2006. http://hdl.handle.net/1721.1/37106.
Full textAbstract:
Thesis (M. Eng.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2006.Includes bibliographical references (p. 115-117).
In this thesis, we develop SignalDB, a framework for composing signal processing applications from primitive stream and signal processing operators. SignalDB allows the user to focus on the signal processing task and avoid needlessly spending time on learning a particular application programming interface (API). We use SignalDB to express acoustic and pressure transient methods for water pipeline monitoring as query plans consisting of signal processing operators.
by Timur Tokmouline.
M.Eng.
5
Robakowski, Mikolaj. "Comparison of State Backends for Modern Stream Processing System." Thesis, KTH, Skolan för elektroteknik och datavetenskap (EECS), 2021. http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-290597.
Full textAbstract:
Distributed Stream Processing is a very popular computing paradigm used invarious modern computer systems. An important aspect of distributed streamprocessing systems is how they deal with computation state bigger than thesystem memory. This is often solved by the usage of a state backend – adatabase, usually an embedded one, that manages the state on the persistentstorage. However, this makes the performance of the whole system dependanton the performance of the database under the given workload. Log-structuredmerge-tree-based solutions are commonly used in stream processing systemsas a one-size-fits-all state backends. We postulate that using different statebackends for different workloads yields much better performance. In this workwe implement several state backends for Arcon, a modern stream processingruntime written in Rust and developed at KTH. The thesis goes over the designchoices and implementation process of a state backend interface alongwith several concrete implementations. We experimentally evaluate the implementationsagainst each other and show that under certain workloads someperform better than other. In particular we show that under read-heavy workloadssled, an embedded Bw-tree-based database written in Rust, outperformsthe commonly used, LSM-based RocksDB.Distribuerad strömbehandling är ett mycket populärt dataparadigm som användsi olika moderna datorsystem. En viktig aspekt av distribuerad strömbearbetningssystem är hur de hanterar data som är större än system minne.Detta löses ofta genom användning av en backend – en databas, vanligtvis eninbäddad, som hanterar lagringen. Detta gör dock att hela systemets prestandablir beroende av databasens prestanda för den angivna arbetsbelastningen.Loggstrukturerad merge-tree-baserade lösningar används ofta i strömbehandlingssystemsom en backend för alla typer av belastningar. Vi postulerar attanvända olika backends för olika arbetsbelastningar ger mycket bättre prestanda.I det här arbetet implementerar vi flera backends för Arcon, en modernströmbehandlings runtime skriven i Rust och utvecklad vid KTH. Avhandlingengår över implementeringsprocessen och gränssnittet för backends med flerakonkreta implementationer. Vi utvärderar experimentellt implementationernamot varandra och visar att vissa presterar bättre än andra beroende på arbetsbelastningen.I synnerhet visar vi att under läs-tungt arbete, så ser vi att sled,en inbäddad Bw-Tree databas skriven i Rust presterar bättre än den vanligaLSM-baserade RocksDB.
6
Mousavi, Bamdad. "Scalable Stream Processing and Management for Time Series Data." Thesis, Université d'Ottawa / University of Ottawa, 2021. http://hdl.handle.net/10393/42295.
Full textAbstract:
There has been an enormous growth in the generation of time series data in the past decade. This trend is caused by widespread adoption of IoT technologies, the data generated by monitoring of cloud computing resources, and cyber physical systems. Although time series data have been a topic of discussion in the domain of data management for several decades, this recent growth has brought the topic to the forefront. Many of the time series management systems available today lack the necessary features to successfully manage and process the sheer amount of time series being generated today. In this today we stive to examine the field and study the prior work in time series management. We then propose a large system capable of handling time series management end to end, from generation to consumption by the end user. Our system is composed of open-source data processing frameworks. Our system has the capability to collect time series data, perform stream processing over it, store it for immediate and future processing and create necessary visualizations. We present the implementation of the system and perform experimentations to show its scalability to handle growing pipelines of incoming data from various sources.
7
Balazinska, Magdalena. "Fault-tolerance and load management in a distributed stream processing system." Thesis, Massachusetts Institute of Technology, 2005. http://hdl.handle.net/1721.1/35287.
Full textAbstract:
Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, February 2006.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.
Includes bibliographical references (p. 187-199).
Advances in monitoring technology (e.g., sensors) and an increased demand for online information processing have given rise to a new class of applications that require continuous, low-latency processing of large-volume data streams. These "stream processing applications" arise in many areas such as sensor-based environment monitoring, financial services, network monitoring, and military applications. Because traditional database management systems are ill-suited for high-volume, low-latency stream processing, new systems, called stream processing engines (SPEs), have been developed. Furthermore, because stream processing applications are inherently distributed, and because distribution can improve performance and scalability, researchers have also proposed and developed distributed SPEs. In this dissertation, we address two challenges faced by a distributed SPE: (1) faulttolerant operation in the face of node failures, network failures, and network partitions, and (2) federated load management. For fault-tolerance, we present a replication-based scheme, called Delay, Process, and Correct (DPC), that masks most node and network failures.
(cont.) When network partitions occur, DPC addresses the traditional availability-consistency trade-off by maintaining, when possible, a desired availability specified by the application or user, but eventually also delivering the correct results. While maintaining the desired availability bounds, DPC also strives to minimize the number of inaccurate results that must later be corrected. In contrast to previous proposals for fault tolerance in SPEs, DPC simultaneously supports a variety of applications that differ in their preferred trade-off between availability and consistency. For load management, we present a Bounded-Price Mechanism (BPM) that enables autonomous participants to collaboratively handle their load without individually owning the resources necessary for peak operation. BPM is based on contracts that participants negotiate offline. At runtime, participants move load only to partners with whom they have a contract and pay each other the contracted price. We show that BPM provides incentives that foster participation and leads to good system-wide load distribution. In contrast to earlier proposals based on computational economies, BPM is lightweight, enables participants to develop and exploit preferential relationships, and provides stability and predictability.
(cont.) Although motivated by stream processing, BPM is general and can be applied to any federated system. We have implemented both schemes in the Borealis distributed stream processing engine. They will be available with the next release of the system.
by Magdalena Balazinska.
Ph.D.
8
Ahmed, Abdulbasit. "Online network intrusion detection system using temporal logic and stream data processing." Thesis, University of Liverpool, 2013. http://livrepository.liverpool.ac.uk/12153/.
Full textAbstract:
These days, the world is becoming more interconnected, and the Internet has dominated the ways to communicate or to do business. Network security measures must be taken to protect the organization environment. Among these security measures are the intrusion detection systems. These systems aim to detect the actions that attempt to compromise the confidentiality, availability, and integrity of a resource by monitoring the events occurring in computer systems and/or networks. The increasing amounts of data that are transmitted at higher and higher speed networks created a challenging problem for the current intrusion detection systems. Once the traffic exceeds the operational boundaries of these systems, packets are dropped. This means that some attacks will not be detected. In this thesis, we propose developing an online network based intrusion detection system by the combined use of temporal logic and stream data processing. Temporal Logic formalisms allow us to represent attack patterns or normal behaviour. Stream data processing is a recent database technology applied to flows of data. It is designed with high performance features for data intensive applications processing. In this work we develop a system where temporal logic specifications are automatically translated into stream queries that run on the stream database server and are continuously evaluated against the traffic to detect intrusions. The experimental results show that this combination was efficient in using the resources of the running machines and was able to detect all the attacks in the test data. Additionally, the proposed solution provides a concise and unambiguous way to formally represent attack signatures and it is extensible allowing attacks to be added. Also, it is scalable as the system can benefit from using more CPUs and additional memory on the same machine, or using distributed servers.
9
Al-Sinayyid, Ali. "JOB SCHEDULING FOR STREAMING APPLICATIONS IN HETEROGENEOUS DISTRIBUTED PROCESSING SYSTEMS." OpenSIUC, 2020. https://opensiuc.lib.siu.edu/dissertations/1868.
Full textAbstract:
The colossal amounts of data generated daily are increasing exponentially at a never-before-seen pace. A variety of applications—including stock trading, banking systems, health-care, Internet of Things (IoT), and social media networks, among others—have created an unprecedented volume of real-time stream data estimated to reach billions of terabytes in the near future. As a result, we are currently living in the so-called Big Data era and witnessing a transition to the so-called IoT era. Enterprises and organizations are tackling the challenge of interpreting the enormous amount of raw data streams to achieve an improved understanding of data, and thus make efficient and well-informed decisions (i.e., data-driven decisions). Researchers have designed distributed data stream processing systems that can directly process data in near real-time. To extract valuable information from raw data streams, analysts need to create and implement data stream processing applications structured as a directed acyclic graphs (DAG). The infrastructure of distributed data stream processing systems, as well as the various requirements of stream applications, impose new challenges. Cluster heterogeneity in a distributed environment results in different cluster resources for task execution and data transmission, which make the optimal scheduling algorithms an NP-complete problem. Scheduling streaming applications plays a key role in optimizing system performance, particularly in maximizing the frame-rate, or how many instances of data sets can be processed per unit of time. The scheduling algorithm must consider data locality, resource heterogeneity, and communicational and computational latencies. The latencies associated with the bottleneck from computation or transmission need to be minimized when mapped to the heterogeneous and distributed cluster resources. Recent work on task scheduling for distributed data stream processing systems has a number of limitations. Most of the current schedulers are not designed to manage heterogeneous clusters. They also lack the ability to consider both task and machine characteristics in scheduling decisions. Furthermore, current default schedulers do not allow the user to control data locality aspects in application deployment.In this thesis, we investigate the problem of scheduling streaming applications on a heterogeneous cluster environment and develop the maximum throughput scheduler algorithm (MT-Scheduler) for streaming applications. The proposed algorithm uses a dynamic programming technique to efficiently map the application topology onto a heterogeneous distributed system based on computing and data transfer requirements, while also taking into account the capacity of underlying cluster resources. The proposed approach maximizes the system throughput by identifying and minimizing the time incurred at the computing/transfer bottleneck. The MT-Scheduler supports scheduling applications that are structured as a DAG, such as Amazon Timestream, Google Millwheel, and Twitter Heron. We conducted experiments using three Storm microbenchmark topologies in both simulated and real Apache Storm environments. To evaluate performance, we compared the proposed MT-Scheduler with the simulated round-robin and the default Storm scheduler algorithms. The results indicated that the MT-Scheduler outperforms the default round-robin approach in terms of both average system latency and throughput.
10
Addimando, Alessio. "Progettazione di un intrusion detection system su piattaforma big data." Master's thesis, Alma Mater Studiorum - Università di Bologna, 2018. http://amslaurea.unibo.it/16755/.
Full textAbstract:
Negli ultimi anni, nel panorama digitale, è stato rilevato un ingente aumento del numero di dispositivi e utenti con accesso ad Internet.
Proporzionalmente a questi fattori ogni giorno vengono generati continuamente, e in qualsiasi contesto, grandi quantità di dati difficili da gestire. Questo ha fatto emergere la necessità di riorganizzare gli asset aziendali per far fronte ad un calibro di informazione maggiore e per far in modo che la gestione stessa ne estragga valore concreto per la realtà decisionale.
L'insieme di queste motivazioni da vita al fenomeno dei Big Data.
Affiancato a questo panorama, inoltre, la grande quantità macchine e utenti in rete ha esponenzialmente aumentato anche il numero di attacchi informatici, che puntano nella stragrande dei casi all'appropriazione non autorizzata di dati sensibili e/o a provocare disservizi nelle reti private.
Un esempio è il campus universitario di Forlì-Cesena che stima costantemente attive circa 3000 macchine interconnesse tra di loro e con la rete esterna.
La grande quantità di risorse connesse in rete assume una certa importanza visti i dati sensibili che gestiscono e immagazzinano e nonostante l'archittettura di monitoraggio venga continuamente aggiornata, quest'ultima presenta colli di bottiglia evidenti e limitazioni nell'elaborazione dell'intero traffico di rete.
Per far fronte a questa problematica lo scopo della tesi è stato quello di far convergere questi due ambiti informatici integrando al processo di sicurezza della rete un sistema di analisi e monitoraggio per il rilevamento di intrusioni (intrusion detection system), su piattaforma Big Data.
Il prototipo realizzato (denominato Styx), sfrutta tecniche di data stream processing (elaborazione di dati real-time) e di machine learning (tecniche di apprendimento per estrazione di modelli predittivi) per potenziare l'attuale sistema di monitoraggio della rete universitaria.
Books on the topic "Stream Processing System"
1
Fredericks, Jeffrey W. Decision support system for conjunctive stream-aquifer management. Fort Collins, Colo: Colorado Water Resources Research Institute, 1995.
Find full text
2
Babbitt, Ronald E. Improved streamflow and water quality monitoring using a microprocessor-based system. Ogden, UT]: U.S. Dept. of Agriculture, Forest Service, Intermountain Research Station, 1987.
Find full text
3
Babbitt, Ronald E. Improved streamflow and water quality monitoring using a microprocessor-based system. [Ogden, UT]: U.S. Dept. of Agriculture, Forest Service, Intermountain Research Station, 1987.
Find full text
4
Datacasting: How to stream databases over the Internet. New York: McGraw-Hill, 1998.
Find full text
5
IFIP World Computer Congress (17th 2002 Montreál, Québec). Intelligent information processing: IFIP 17th World Computer Congress-TC12 stream on intelligent information processing, August 25-30, 2002, Montreál, Québec, Canada. Boston: Kluwer Academic, 2002.
Find full text
6
Eric, Yen, and ebrary Inc, eds. Oracle 11g Streams implementer's guide: Design, implement, and maintain a distributed environment with Oracle Streams. Birmingham, U.K: Packt Pub., 2010.
Find full text
7
Corporation, International Business Machines, ed. IBM Infosphere Streams harnessing data in motion. [S.l.]: Vervante, 2010.
Find full text
8
Lane, Norman E. Users manual for the Automated Performance Test System (APTS). Orlando, FL: Essex Corp., 1990.
Find full text
9
A, Hemstrom Miles, and Pacific Northwest Research Station (Portland, Or.), eds. Midscale analysis of streamside characteristics in the upper Grande Ronde subbasin, northeastern Oregon. Portland, OR: U.S. Dept. of Agriculture, Forest Service, Pacific Northwest Research Station, 2002.
Find full text
10
Krivoyekov, Syergyey, and Roman Ayzman. Psychophysiology. ru: INFRA-M Academic Publishing LLC., 2015. http://dx.doi.org/10.12737/10884.
Full textAbstract:
Psychophysiology — the science studying interrelation of mentality of the person and physiological processes. Fundamental knowledge of work of a brain, first of all, of nervous regulation of functions of an organism, the general and specific features of the highest falls within the scope of its interests
nervous activity, the defining character and behavior of the person,
psychophysiological mechanisms of regulation of functional states.
In the book neurophysiological bases of coding and information processing in nervous system, neural mechanisms of feelings, perceptions, memories, training, motivations and emotions, thinking and the speech, attention, consciousness, behavior, mental activity are stated. Separate
the section is devoted to physiological bases of mental changes at various functional, extreme and pathophysiological states (a stress, post-stressful frustration, addiktivny states, depressions, etc.) and to ways of their correction. Authors tried to pay special attention to disclosure of specifics of psychophysiology of the person,
to difference of physiological mechanisms of regulation of mental functions of the person in comparison with model researches on animals.
For simplification of work on discipline and the best digestion of material
the textbook is supplied with the glossary.
For students, undergraduates, graduate students and teachers of psychological and medical faculties of higher education institutions.
Book chapters on the topic "Stream Processing System"
1
Gorawski, Marcin, Pawel Marks, and Michal Gorawski. "Modeling Data Stream Intensity in Distributed Stream Processing System." In Computer Networks, 372–83. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013. http://dx.doi.org/10.1007/978-3-642-38865-1_38.
Full text
2
Gilani, Altaf, Satyajeet Sonune, Balakumar Kendai, and Sharma Chakravarthy. "The Anatomy of a Stream Processing System." In Flexible and Efficient Information Handling, 232–39. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006. http://dx.doi.org/10.1007/11788911_20.
Full text
3
Le, Jia-jin, and Jian-wei Liu. "DDSQP: A WSRF-Based Distributed Data Stream Query System." In Parallel and Distributed Processing and Applications, 833–44. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005. http://dx.doi.org/10.1007/11576235_83.
Full text
4
Zrilic, Djuro G. "A Δ-Σ Digital Amplitude Modulation System." In Functional Processing of Delta-Sigma Bit-Stream, 75–82. Cham: Springer International Publishing, 2020. http://dx.doi.org/10.1007/978-3-030-47648-9_8.
Full text
5
Nishii, Shunsuke, and Toyotaro Suzumura. "Highly Scalable Speech Processing on Data Stream Management System." In Database Systems for Advanced Applications, 203–12. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012. http://dx.doi.org/10.1007/978-3-642-29035-0_14.
Full text
6
Wang, Xiaotong, Junhua Fang, Yuming Li, Rong Zhang, and Aoying Zhou. "Cost-Effective Data Partition for Distributed Stream Processing System." In Database Systems for Advanced Applications, 623–35. Cham: Springer International Publishing, 2017. http://dx.doi.org/10.1007/978-3-319-55699-4_39.
Full text
7
Zrilic, Djuro G. "A Δ-Σ Digital Stereo Multiplexing–Demultiplexing System." In Functional Processing of Delta-Sigma Bit-Stream, 67–74. Cham: Springer International Publishing, 2020. http://dx.doi.org/10.1007/978-3-030-47648-9_7.
Full text
8
Zou, Beiji, Tao Zhang, Chengzhang Zhu, Ling Xiao, Meng Zeng, and Zhi Chen. "Alps: An Adaptive Load Partitioning Scaling Solution for Stream Processing System on Skewed Stream." In Lecture Notes in Computer Science, 17–31. Cham: Springer International Publishing, 2022. http://dx.doi.org/10.1007/978-3-031-12426-6_2.
Full text
9
Chakravarthy, Sharma, and Qingchun Jiang. "NFMi: AN INTER-DOMAIN NETWORK FAULT MANAGEMENT SYSTEM." In Stream Data Processing: A Quality of Service Perspective, 167–86. Boston, MA: Springer US, 2009. http://dx.doi.org/10.1007/978-0-387-71003-7_8.
Full text
10
Jiang, Jiawei, Zhipeng Zhang, Bin Cui, Yunhai Tong, and Ning Xu. "StroMAX: Partitioning-Based Scheduler for Real-Time Stream Processing System." In Database Systems for Advanced Applications, 269–88. Cham: Springer International Publishing, 2017. http://dx.doi.org/10.1007/978-3-319-55699-4_17.
Full textConference papers on the topic "Stream Processing System"
1
Park, Alfred J., Cheng-Hong Li, Ravi Nair, Nobuyuki Ohba, Uzi Shvadron, Ayal Zaks, and Eugen Schenfeld. "Flow: A Stream Processing System Simulator." In 2010 IEEE 24th Workshop on Principles of Advanced and Distributed Simulation (PADS 2010). IEEE, 2010. http://dx.doi.org/10.1109/pads.2010.5471658.
Full text
2
Lee, Myungcheol, Miyoung Lee, Sung Jin Hur, and Ikkyun Kim. "Load adaptive distributed stream processing system for explosive stream data." In 2015 17th International Conference on Advanced Communication Technology (ICACT). IEEE, 2015. http://dx.doi.org/10.1109/icact.2015.7224896.
Full text
3
Kwon, Oje, Yong-Soo Song, Jae-Hun Kim, and Ki-Joune Li. "SCONSTREAM: A Spatial Context Stream Processing System." In 2010 International Conference on Computational Science and Its Applications. IEEE, 2010. http://dx.doi.org/10.1109/iccsa.2010.48.
Full text
4
Körber, Michael, Jakob Eckstein, Nikolaus Glombiewski, and Bernhard Seeger. "Event Stream Processing on Heterogeneous System Architecture." In the 15th International Workshop. New York, New York, USA: ACM Press, 2019. http://dx.doi.org/10.1145/3329785.3329933.
Full text
5
Michalak, Peter, and Paul Watson. "PATH2iot: A Holistic, Distributed Stream Processing System." In 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom). IEEE, 2017. http://dx.doi.org/10.1109/cloudcom.2017.35.
Full text
6
Wladdimiro, Daniel, Luciana Arantes, Pierre Sens, and Nicolas Hidalgo. "A Multi-Metric Adaptive Stream Processing System." In 2021 IEEE 20th International Symposium on Network Computing and Applications (NCA). IEEE, 2021. http://dx.doi.org/10.1109/nca53618.2021.9685871.
Full text
7
"ITAIPU DATA STREAM MANAGEMENT SYSTEM - A Stream Processing System with Business Users in Mind." In 3rd International Conference on Software and Data Technologies. SciTePress - Science and and Technology Publications, 2008. http://dx.doi.org/10.5220/0001882000540064.
Full text
8
Wang, Shengxiang, Hansheng Lu, Zhiyun Gao, and Shanfeng Hou. "Multifunctional video stream processing system based on DSP." In Photonics Asia 2002, edited by LiWei Zhou, Chung-Sheng Li, and Yoshiji Suzuki. SPIE, 2002. http://dx.doi.org/10.1117/12.481567.
Full text
9
Alves de Souza Ramos, Thatyene Louise, Rodrigo Silva Oliveira, Ana Paula de Carvalho, Renato Antonio Celso Ferreira, and Wagner Meira Jr. "Watershed: A High Performance Distributed Stream Processing System." In 2011 23rd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD). IEEE, 2011. http://dx.doi.org/10.1109/sbac-pad.2011.31.
Full text
10
Vogler, Michael, Johannes M. Schleicher, Christian Inzinger, Bernhard Nickel, and Schahram Dustdar. "Non-intrusive Monitoring of Stream Processing Applications." In 2016 IEEE Symposium on Service-Oriented System Engineering (SOSE). IEEE, 2016. http://dx.doi.org/10.1109/sose.2016.11.
Full textReports on the topic "Stream Processing System"
1
Alchanatis, Victor, Stephen W. Searcy, Moshe Meron, W. Lee, G. Y. Li, and A. Ben Porath. Prediction of Nitrogen Stress Using Reflectance Techniques. United States Department of Agriculture, November 2001. http://dx.doi.org/10.32747/2001.7580664.bard.
Full textAbstract:
Commercial agriculture has come under increasing pressure to reduce nitrogen fertilizer inputs in order to minimize potential nonpoint source pollution of ground and surface waters. This has resulted in increased interest in site specific fertilizer management. One way to solve pollution problems would be to determine crop nutrient needs in real time, using remote detection, and regulating fertilizer dispensed by an applicator. By detecting actual plant needs, only the additional nitrogen necessary to optimize production would be supplied. This research aimed to develop techniques for real time assessment of nitrogen status of corn using a mobile sensor with the potential to regulate nitrogen application based on data from that sensor. Specifically, the research first attempted to determine the system parameters necessary to optimize reflectance spectra of corn plants as a function of growth stage, chlorophyll and nitrogen status. In addition to that, an adaptable, multispectral sensor and the signal processing algorithm to provide real time, in-field assessment of corn nitrogen status was developed. Spectral characteristics of corn leaves reflectance were investigated in order to estimate the nitrogen status of the plants, using a commercial laboratory spectrometer. Statistical models relating leaf N and reflectance spectra were developed for both greenhouse and field plots. A basis was established for assessing nitrogen status using spectral reflectance from plant canopies. The combined effect of variety and N treatment was studied by measuring the reflectance of three varieties of different leaf characteristic color and five different N treatments. The variety effect on the reflectance at 552 nm was not significant (a = 0.01), while canonical discriminant analysis showed promising results for distinguishing different variety and N treatment, using spectral reflectance. Ambient illumination was found inappropriate for reliable, one-beam spectral reflectance measurement of the plants canopy due to the strong spectral lines of sunlight. Therefore, artificial light was consequently used. For in-field N status measurement, a dark chamber was constructed, to include the sensor, along with artificial illumination. Two different approaches were tested (i) use of spatially scattered artificial light, and (ii) use of collimated artificial light beam. It was found that the collimated beam along with a proper design of the sensor-beam geometry yielded the best results in terms of reducing the noise due to variable background, and maintaining the same distance from the sensor to the sample point of the canopy. A multispectral sensor assembly, based on a linear variable filter was designed, constructed and tested. The sensor assembly combined two sensors to cover the range of 400 to 1100 nm, a mounting frame, and a field data acquisition system. Using the mobile dark chamber and the developed sensor, as well as an off-the-shelf sensor, in- field nitrogen status of the plants canopy was measured. Statistical analysis of the acquired in-field data showed that the nitrogen status of the com leaves can be predicted with a SEP (Standard Error of Prediction) of 0.27%. The stage of maturity of the crop affected the relationship between the reflectance spectrum and the nitrogen status of the leaves. Specifically, the best prediction results were obtained when a separate model was used for each maturity stage. In-field assessment of the nitrogen status of corn leaves was successfully carried out by non contact measurement of the reflectance spectrum. This technology is now mature to be incorporated in field implements for on-line control of fertilizer application.
2
Christopher, David A., and Avihai Danon. Plant Adaptation to Light Stress: Genetic Regulatory Mechanisms. United States Department of Agriculture, May 2004. http://dx.doi.org/10.32747/2004.7586534.bard.
Full textAbstract:
Original Objectives: 1. Purify and biochemically characterize RB60 orthologs in higher plant chloroplasts; 2. Clone the gene(s) encoding plant RB60 orthologs and determine their structure and expression; 3. Manipulate the expression of RB60; 4. Assay the effects of altered RB60 expression on thylakoid biogenesis and photosynthetic function in plants exposed to different light conditions. In addition, we also examined the gene structure and expression of RB60 orthologs in the non-vascular plant, Physcomitrella patens and cloned the poly(A)-binding protein orthologue (43 kDa RB47-like protein). This protein is believed to a partner that interacts with RB60 to bind to the psbA5' UTR. Thus, to obtain a comprehensive view of RB60 function requires analysis of its biochemical partners such as RB43. Background & Achievements: High levels of sunlight reduce photosynthesis in plants by damaging the photo system II reaction center (PSII) subunits, such as D1 (encoded by the chloroplast tpsbAgene). When the rate of D1 synthesis is less than the rate of photo damage, photo inhibition occurs and plant growth is decreased. Plants use light-activated translation and enhanced psbAmRNA stability to maintain D1 synthesis and replace the photo damaged 01. Despite the importance to photosynthetic capacity, these mechanisms are poorly understood in plants. One intriguing model derived from the algal chloroplast system, Chlamydomonas, implicates the role of three proteins (RB60, RB47, RB38) that bind to the psbAmRNA 5' untranslated leader (5' UTR) in the light to activate translation or enhance mRNA stability. RB60 is the key enzyme, protein D1sulfide isomerase (Pill), that regulates the psbA-RN :Binding proteins (RB's) by way of light-mediated redox potentials generated by the photosystems. However, proteins with these functions have not been described from higher plants. We provided compelling evidence for the existence of RB60, RB47 and RB38 orthologs in the vascular plant, Arabidopsis. Using gel mobility shift, Rnase protection and UV-crosslinking assays, we have shown that a dithiol redox mechanism which resembles a Pill (RB60) activity regulates the interaction of 43- and 30-kDa proteins with a thermolabile stem-loop in the 5' UTR of the psbAmRNA from Arabidopsis. We discovered, in Arabidopsis, the PD1 gene family consists of II members that differ in polypeptide length from 361 to 566 amino acids, presence of signal peptides, KDEL motifs, and the number and positions of thioredoxin domains. PD1's catalyze the reversible formation an disomerization of disulfide bonds necessary for the proper folding, assembly, activity, and secretion of numerous enzymes and structural proteins. PD1's have also evolved novel cellular redox functions, as single enzymes and as subunits of protein complexes in organelles. We provide evidence that at least one Pill is localized to the chloroplast. We have used PDI-specific polyclonal and monoclonal antisera to characterize the PD1 (55 kDa) in the chloroplast that is unevenly distributed between the stroma and pellet (containing membranes, DNA, polysomes, starch), being three-fold more abundant in the pellet phase. PD1-55 levels increase with light intensity and it assembles into a high molecular weight complex of ~230 kDa as determined on native blue gels. In vitro translation of all 11 different Pill's followed by microsomal membrane processing reactions were used to differentiate among PD1's localized in the endoplasmic reticulum or other organelles. These results will provide.1e insights into redox regulatory mechanisms involved in adaptation of the photosynthetic apparatus to light stress. Elucidating the genetic mechanisms and factors regulating chloroplast photosynthetic genes is important for developing strategies to improve photosynthetic efficiency, crop productivity and adaptation to high light environments.
3
Ron, Eliora, and Eugene Eugene Nester. Global functional genomics of plant cell transformation by agrobacterium. United States Department of Agriculture, March 2009. http://dx.doi.org/10.32747/2009.7695860.bard.
Full textAbstract:
The aim of this study was to carry out a global functional genomics analysis of plant cell transformation by Agrobacterium in order to define and characterize the physiology of Agrobacterium in the acidic environment of a wounded plant. We planed to study the proteome and transcriptome of Agrobacterium in response to a change in pH, from 7.2 to 5.5 and identify genes and circuits directly involved in this change. Bacteria-plant interactions involve a large number of global regulatory systems, which are essential for protection against new stressful conditions. The interaction of bacteria with their hosts has been previously studied by genetic-physiological methods. We wanted to make use of the new capabilities to study these interactions on a global scale, using transcription analysis (transcriptomics, microarrays) and proteomics (2D gel electrophoresis and mass spectrometry). The results provided extensive data on the functional genomics under conditions that partially mimic plant infection and – in addition - revealed some surprising and significant data. Thus, we identified the genes whose expression is modulated when Agrobacterium is grown under the acidic conditions found in the rhizosphere (pH 5.5), an essential environmental factor in Agrobacterium – plant interactions essential for induction of the virulence program by plant signal molecules. Among the 45 genes whose expression was significantly elevated, of special interest is the two-component chromosomally encoded system, ChvG/I which is involved in regulating acid inducible genes. A second exciting system under acid and ChvG/Icontrol is a secretion system for proteins, T6SS, encoded by 14 genes which appears to be important for Rhizobium leguminosarum nodule formation and nitrogen fixation and for virulence of Agrobacterium. The proteome analysis revealed that gamma aminobutyric acid (GABA), a metabolite secreted by wounded plants, induces the synthesis of an Agrobacterium lactonase which degrades the quorum sensing signal, N-acyl homoserine lactone (AHL), resulting in attenuation of virulence. In addition, through a transcriptomic analysis of Agrobacterium growing at the pH of the rhizosphere (pH=5.5), we demonstrated that salicylic acid (SA) a well-studied plant signal molecule important in plant defense, attenuates Agrobacterium virulence in two distinct ways - by down regulating the synthesis of the virulence (vir) genes required for the processing and transfer of the T-DNA and by inducing the same lactonase, which in turn degrades the AHL. Thus, GABA and SA with different molecular structures, induce the expression of these same genes. The identification of genes whose expression is modulated by conditions that mimic plant infection, as well as the identification of regulatory molecules that help control the early stages of infection, advance our understanding of this complex bacterial-plant interaction and has immediate potential applications to modify it. We expect that the data generated by our research will be used to develop novel strategies for the control of crown gall disease. Moreover, these results will also provide the basis for future biotechnological approaches that will use genetic manipulations to improve bacterial-plant interactions, leading to more efficient DNA transfer to recalcitrant plants and robust symbiosis. These advances will, in turn, contribute to plant protection by introducing genes for resistance against other bacteria, pests and environmental stress.
4
Steffens, John C., and Eithan Harel. Polyphenol Oxidases- Expression, Assembly and Function. United States Department of Agriculture, January 1995. http://dx.doi.org/10.32747/1995.7571358.bard.
Full textAbstract:
Polyphenol oxidases (PPOs) participate in the preparation of many plant products on the one hand and cause considerable losses during processing of plant products on the other hand. However, the physiological functions of plant PPO were still a subject of controversy at the onset of the project. Preliminary observations that suggested involvement of PPOs in resistance to herbivores and pathogens held great promise for application in agriculture but required elucidation of PPO's function if modulation of PPO expression is to be considered for improving plant protection or storage and processing of plant products. Suggestions on a possible role of PPO in various aspects of chloroplast metabolism were also relevant in this context. The characterization of plant PPO genes opened a way for achieving these goals. We reasoned that "understanding PPO targeting and routing, designing ways to manipulate its expression and assessing the effects of such modifications will enable determination of the true properties of the enzyme and open the way for controlling its activity". The objective of the project was to "obtain an insight into the function and biological significance of PPOs" by examining possible function(s) of PPO in photosynthesis and plant-pest interactions using transgenic tomato plants; extending our understanding of PPO routing and assembly and the mechanism of its thylakoid translocation; preparing recombinant PPOs for use in import studies, determination of the genuine properties of PPOs and understanding its assembly and determining the effect of PPO's absence on chloroplast performance. Results obtained during work on the project made it necessary to abandon some minor objectives and devote the effort to more promising topics. Such changes are mentioned in the 'Body of the report' which is arranged according to the objectives of the original proposal. The complex expression pattern of tomato PPO gene family was determined. Individual members of the family are differentially expressed in various parts of the plant and subjected to developmentally regulated turnover. Some members are differentially regulated also by pathogens, wounding and chemical wound signals. Wounding systemically induces PPO activity and level in potato. Only tissues that are developmentally competent to express PPO are capable of responding to the systemic wounding signal by increased accumulation of PPO mRNA. Down regulation of PPO genes causes hyper susceptibility to leaf pathogens in tomato while over expression regulation of PPO expression in tomato plants is their apparent increased tolerance to drought. Both the enhanced disease resistance conferred by PPO over expression and the increased stress tolerance due to down regulation can be used in the engineering of improved crop plants. Photosynthesis rate and variable fluorescence measurements in wild type, and PPO-null and over expressing transgenic tomato lines suggest that PPO does not enable plants to cope better with stressful high light intensities or reactive oxygen species. Rather high levels of the enzyme aggravate the damage caused under such conditions. Our work suggests that PPO's primary role is in defending plants against pathogens and herbivores. Jasmonate and ethylene, and apparently also salicylate, signals involved in responses to wounding and defense against herbivores and pathogens, enhance markedly and specifically the competence of chloroplasts to import and process pPPO. The interaction of the precursor with thylakoid membranes is primarily affected. The routing of PPO shows other unusual properties: stromal processing occurs in two sites, resulting in intermediates that are translocated across thylakoids by two different mechanisms - a DpH- and a Sec-dependent one. It is suggested that the dual pattern of processing and routing constitutes a'fail safe' mechanism, reflecting the need for a rapid and flexible response to defense challenges. Many of the observations described above should be taken into consideration when manipulation of PPO expression is contemplated for use in crop improvement.