QUAY: A data stream processing system using chunking

Ken C.K. Lee, Hong Va Leong, Antonio Si

Research output: Journal article publicationConference articleAcademic researchpeer-review

1 Citation (Scopus)

Abstract

Data stream processing has emerged as a recent research direction focusing on new generation database applications, in which data records from remote source sites flow continuously to a processing site. Queries residing in the processing site are triggered and evaluated upon the arrival of their interested data records. There are two important aspects that distinguish data stream processing systems from conventional database, systems. First, the roles of queries and data records are swapped; queries are stationary while data records are dynamic. Query indexing becomes an essential performance determining issue. Second, the expectedly high data flow rate aggravates data index maintenance overheads. To address the problems thus arisen, we propose and develop a data stream processing system called QUAY. In this paper, we present the design, implementation and evaluation of QUAY. The core technique that we use is "chunking" which clusters and indexes both queries and data records in a unified way as chunks. To process window join operation from stream sources, we propose an adaptive selection-join arrangement for a huge number of selection-join queries to share expensive join operations. Through a set of intensive performance evaluation experiments, we show that the chunking organization, operating under our proposed adaptive selection-join arrangement, yields desirably good performance.
Original languageEnglish
Pages (from-to)17-26
Number of pages10
JournalProceedings of the International Database Engineering and Applications Symposium, IDEAS
Publication statusPublished - 25 Oct 2004
EventProceedings - International Database Engineering and Applications Symposium, IDEAS'04 - Coimbra, Portugal
Duration: 7 Jul 20049 Jul 2004

ASJC Scopus subject areas

  • Computer Science(all)
  • Engineering(all)

Cite this