Towards efficient inference: Adaptively cooperate in heterogeneous IoT Edge Cluster

Xiang Yang, Qi Qi, Jingyu Wang, Song Guo, Jianxin Liao

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

Abstract

New applications such as smart homes, autonomous vehicles are leading an increasing research topic of convolutional neural network (CNN) based inference on IoT edge devices. Unfortunately, this scenario meets a huge roadblock caused by the limited computing resources owned by these devices. One popular solution is to execute inference on an edge cluster with parallelization schemes instead of on a single device. However, the heterogeneous edge devices and varied neural layers bring challenges to this process. In this paper, we propose a pipelined cooperation scheme (PICO) to efficiently execute CNN inference for edge devices. Our goal is to maximize throughput by reducing redundant computing meanwhile to keep the inference latency under a certain value. PICO divides the neural layers and edge devices into several stages. The input data is fed into the first stage and the inference result is produced at the last stage. These stages compose an inference pipeline. The execution time of each stage is optimized to approach the maximum throughput as close as possible. We also implement an adaptive framework to choose the best inference scheme under different workloads. In our experiment with 8 RaspberryPi devices, the average inference latency can be reduced by $1.7\sim 6.5\times$ under different workloads, and the throughput can be improved by $1.8\sim 6.2\times$ under various network settings.

Original languageEnglish
Title of host publicationProceedings - 2021 IEEE 41st International Conference on Distributed Computing Systems, ICDCS 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages12-23
Number of pages12
ISBN (Electronic)9781665445139
DOIs
Publication statusPublished - Jul 2021
Event41st IEEE International Conference on Distributed Computing Systems, ICDCS 2021 - Virtual, Washington, United States
Duration: 7 Jul 202110 Jul 2021

Publication series

NameProceedings - International Conference on Distributed Computing Systems
Volume2021-July

Conference

Conference41st IEEE International Conference on Distributed Computing Systems, ICDCS 2021
Country/TerritoryUnited States
CityVirtual, Washington
Period7/07/2110/07/21

Keywords

  • Edge Computing
  • Model Deployment
  • Pipelined Inference

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'Towards efficient inference: Adaptively cooperate in heterogeneous IoT Edge Cluster'. Together they form a unique fingerprint.

Cite this