Adapting cakeDB to integrate high-pressure big data streams with low-pressure systems

Peter Membrey, Chun Chung Chan, Yuri Demchenko

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

Abstract

Big Data continues to be one of the hottest topics in the computer science field and itself takes many forms. One way Big Data manifests is in the form of streams. These streams can be generally defined by their update frequency and the bandwidth they consume. They can however be further defined by the characteristics of the data they carry. The producers of these streams are generally tuned to perform a given role (such as moving large quantities of data with low latency) which can often be at odds with the requirements of a given consumer. In many cases the logistics of consuming such a stream can make the task impractical. This paper discusses the concept of data streams as sequential data sets and having different pressures. The paper demonstrates through a use case of a financial trading company and a High Performance Compute Cluster how different applications require different pressures and why it is necessary to be able to scale down high pressure streams for low pressure applications without impacting the applications that require the full high pressure feed and the high pressure feed itself. A proposed system for classifying streams and related consumers is discussed as well as the concept of conflation as it applies to these data streams. Features in the prototype stream oriented database (CakeDB) that support adapting high-pressure streams to low-pressure applications are then discussed and further work is identified.
Original languageEnglish
Title of host publicationProceedings - 2013 International Conference on Cloud Computing and Big Data, CLOUDCOM-ASIA 2013
PublisherIEEE Computer Society
Pages414-419
Number of pages6
ISBN (Print)9781479928293
DOIs
Publication statusPublished - 1 Jan 2013
Event2013 International Conference on Cloud Computing and Big Data, CLOUDCOM-ASIA 2013 - Fuzhou, Fujian, China
Duration: 16 Dec 201318 Dec 2013

Conference

Conference2013 International Conference on Cloud Computing and Big Data, CLOUDCOM-ASIA 2013
CountryChina
CityFuzhou, Fujian
Period16/12/1318/12/13

Keywords

  • big data
  • Cakedb
  • Erlang
  • high pressure data streams
  • low latency data streams

ASJC Scopus subject areas

  • Software

Cite this