Method and system for dynamic application composition in streaming systems

Abstract

A system and method for dynamically building applications for stream processing includes providing processing elements with a flow specification describing each input and a stream description describing each output such that the flow specification indicates a stream or streams which are to be received based on processing information and the stream descriptions indicate the processing information. Processing elements that can be reused are identified by determining equivalence between the processing elements. Processing elements that are new and are not reusable are instantiated in a flow graph. An application is dynamically composed, using the instantiated processing elements by routing available streams to the instantiated processing elements in accordance with the flow specifications.

Description

GOVERNMENT RIGHTS [0001] This invention was made with Government support under Contract No.: TIA H98230-04-3-0001 awarded by the U.S. Department of Defense. The Government has certain rights in this invention.BACKGROUND [0002] 1. Technical Field [0003] The present invention relates to systems and methods for processing data streams and more particularly to mechanisms for processing streams of data enabling the dynamic and automatic composition of applications using processing elements (PE) operating on multiple continuous streams of data. The dynamic composition may comprise two tasks (i) determine whether a PE already running in the system can be reused for a new application and (ii) determine streaming connections based on a novel way of specifying streams and flows for PE ports. This specification allows the application writer to tap into streams that are being produced by other applications or will become available in the future. In illustrative examples, these applications are ...

AI Technical Summary

Benefits of technology

The present invention provides a way to automatically route data streams through processing elements (PEs) across applications. This is achieved by dynamically determining which streams connect to PEs based on their flow specifications and stream descriptions. The system / method also allows for the flexible specification of streams and flows, and identifies which PE needs to be instantiated or reused. The invention enables dynamic composition of an application by routing available streams to the instantiated PE in accordance with their flow specifications. The technical effects of the invention include improved routing and flexibility for processing data streams, reduced burden of content-routing specification from the programmer, and efficient use of PE resources.

Problems solved by technology

Traditional information processing techniques, however, are static in nature, in two respects.

This store-and-analyze technique is not suitable for continuous monitoring or for obtaining real-time results because in many cases it is not possible to store all incoming data, and the cost of reprocessing old data can hinder the application performance considerably.

Second, applications are static in terms of the computation or processing applied to the data.

In other words, the computation or analysis does not adapt to new and additional stream data sources being incorporated into the system.

This does not, however, address the requirement of applications that are to be incrementally composed and / or dynamically reassembled from data-processing building blocks in response to changes in the data flowing into the system or new inquiries from analysts over time.

Without a means to enforce this declaration, the content routing becomes ambiguous and the dynamic and transparent composition of operators cannot be achieved.

Because these techniques deal with persistent and static data, they are not adequate to describe flows of streams in a stream-processing context such that individual stages in a computation pipeline can specify their inputs and outputs in terms of attributes and operators.

Images