A cluster based approach for task scheduling across multiple programming systems

Hongliang Lu, Jiannong Cao, Shailey Chawla, Yuqi Wang, Saohe Lv, Xiaodong Wang

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

Abstract

As the data processing demands have been increasing, different types of data processing systems are being developed. The new programming systems have different characteristics like types of data handled, processing technique and performance. However, multiple new systems have introduced difficulties for non-expert users like choosing the right system and usage methodology of the new systems. In order to relieve the burden of common users of conducting data processing tasks and taking relevant advantage of the systems features, we intend to integrate the popular programming systems and provide more efficient data processing services. In this paper, we propose to address the task scheduling problem for integrating multiple programming systems. We have designed a cluster based approach for task scheduling across multiple programming systems. This approach helps in minimizing the makespan of workflows and resource consumption. The simulation results show that the proposed approach can reduce the resource consumption significantly while achieving a low makespan for the workflows.
Original languageEnglish
Title of host publicationProceedings - 15th International Symposium on Parallel and Distributed Computing, ISPDC 2016
PublisherIEEE
Pages222-229
Number of pages8
ISBN (Electronic)9781509041527
DOIs
Publication statusPublished - 18 Apr 2017
Event15th International Symposium on Parallel and Distributed Computing, ISPDC 2016 - Fuzhou, Fujian, China
Duration: 8 Jul 201610 Jul 2016

Conference

Conference15th International Symposium on Parallel and Distributed Computing, ISPDC 2016
Country/TerritoryChina
CityFuzhou, Fujian
Period8/07/1610/07/16

Keywords

  • Big data
  • Cluster based
  • Data processing systems
  • Genetic algorithm
  • Task scheduling

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture
  • Information Systems

Cite this