Data Streams Management In The Real-time Data Warehouse: Functioning Of The Data Streams Processor

Abstract

Data stream applications are currently emerged as sources of Real Time Data Warehouse (RTDW) systems. Existing Extraction, Transformation and Loading (ETL) tools have not been crafted for fast, continuous and time-varying data streams. To address this, we have already proposed a framework of RTDW to efficiently capture process and load the data streams. This article provides technical architecture of Data Streams Processor, the main component of the framework. It discusses approximations in detail including data structures and processes to manage fast data streams in memory. Moreover, algorithm for approximations is presented and clarified with an example. Finally, Data Streams Processor is applied on a real-world case study. (This paper has been extracted from the thesis of Fiaz Majeed, 2009)