Traffic-aware task placement with guaranteed job completion time for geo-distributed big data

Peng Li, Toshiaki Miyazaki, Song Guo

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

1 Citation (Scopus)

Abstract

Big data analysis is usually casted into parallel jobs running on geo-distributed data centers. Different from a single data center, geo-distributed environment imposes big challenges for big data analytics due to the limited network bandwidth between data centers located in different regions. Although research efforts have been devoted to geo-distributed big data, the results are still far from being efficient because of their suboptimal performance or high complexity. In this paper, we propose a traffic-aware task placement to minimize job completion time of big data jobs. We formulate the problem as a non-convex optimization problem and design an algorithm to solve it with proved performance gap. Finally, extensive simulations are conducted to evaluate the performance of our proposal. The simulation results show that our algorithm can reduce job completion time by 40%, compared to a conventional approach that aggregates all data for centralized processing. Meanwhile, it has only 10% performance gap with the optimal solution, but its problem-solving time is extremely small.

Original languageEnglish
Title of host publication2017 IEEE International Conference on Communications, ICC 2017
EditorsMerouane Debbah, David Gesbert, Abdelhamid Mellouk
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781467389990
DOIs
Publication statusPublished - 28 Jul 2017
Event2017 IEEE International Conference on Communications, ICC 2017 - Paris, France
Duration: 21 May 201725 May 2017

Publication series

NameIEEE International Conference on Communications
ISSN (Print)1550-3607

Conference

Conference2017 IEEE International Conference on Communications, ICC 2017
CountryFrance
CityParis
Period21/05/1725/05/17

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Electrical and Electronic Engineering

Cite this