TY - GEN
T1 - Towards efficient inference: Adaptively cooperate in heterogeneous IoT Edge Cluster
AU - Yang, Xiang
AU - Qi, Qi
AU - Wang, Jingyu
AU - Guo, Song
AU - Liao, Jianxin
N1 - Funding Information:
This work was supported in part by the National Key R&D Program of China 2020YFB1807805, in part by the National Natural Science Foundation of China under Grants 62071067, 61771068, 61872310, in part by the Hong Kong RGC Research Impact Fund (RIF) with the Project No. R5060-19 and R5034-18, General Research Fund (GRF) with the Project No. 152221/19E and 15220320/20E, Collaborative Research Fund (CRF) with the Project No. C5026-18G, and Shenzhen Science and Technology Innovation Commission (R2020A045).
Publisher Copyright:
© 2021 IEEE.
PY - 2021/7
Y1 - 2021/7
N2 - New applications such as smart homes, autonomous vehicles are leading an increasing research topic of convolutional neural network (CNN) based inference on IoT edge devices. Unfortunately, this scenario meets a huge roadblock caused by the limited computing resources owned by these devices. One popular solution is to execute inference on an edge cluster with parallelization schemes instead of on a single device. However, the heterogeneous edge devices and varied neural layers bring challenges to this process. In this paper, we propose a pipelined cooperation scheme (PICO) to efficiently execute CNN inference for edge devices. Our goal is to maximize throughput by reducing redundant computing meanwhile to keep the inference latency under a certain value. PICO divides the neural layers and edge devices into several stages. The input data is fed into the first stage and the inference result is produced at the last stage. These stages compose an inference pipeline. The execution time of each stage is optimized to approach the maximum throughput as close as possible. We also implement an adaptive framework to choose the best inference scheme under different workloads. In our experiment with 8 RaspberryPi devices, the average inference latency can be reduced by $1.7\sim 6.5\times$ under different workloads, and the throughput can be improved by $1.8\sim 6.2\times$ under various network settings.
AB - New applications such as smart homes, autonomous vehicles are leading an increasing research topic of convolutional neural network (CNN) based inference on IoT edge devices. Unfortunately, this scenario meets a huge roadblock caused by the limited computing resources owned by these devices. One popular solution is to execute inference on an edge cluster with parallelization schemes instead of on a single device. However, the heterogeneous edge devices and varied neural layers bring challenges to this process. In this paper, we propose a pipelined cooperation scheme (PICO) to efficiently execute CNN inference for edge devices. Our goal is to maximize throughput by reducing redundant computing meanwhile to keep the inference latency under a certain value. PICO divides the neural layers and edge devices into several stages. The input data is fed into the first stage and the inference result is produced at the last stage. These stages compose an inference pipeline. The execution time of each stage is optimized to approach the maximum throughput as close as possible. We also implement an adaptive framework to choose the best inference scheme under different workloads. In our experiment with 8 RaspberryPi devices, the average inference latency can be reduced by $1.7\sim 6.5\times$ under different workloads, and the throughput can be improved by $1.8\sim 6.2\times$ under various network settings.
KW - Edge Computing
KW - Model Deployment
KW - Pipelined Inference
UR - http://www.scopus.com/inward/record.url?scp=85117072029&partnerID=8YFLogxK
U2 - 10.1109/ICDCS51616.2021.00011
DO - 10.1109/ICDCS51616.2021.00011
M3 - Conference article published in proceeding or book
AN - SCOPUS:85117072029
T3 - Proceedings - International Conference on Distributed Computing Systems
SP - 12
EP - 23
BT - Proceedings - 2021 IEEE 41st International Conference on Distributed Computing Systems, ICDCS 2021
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 41st IEEE International Conference on Distributed Computing Systems, ICDCS 2021
Y2 - 7 July 2021 through 10 July 2021
ER -