FedDM: Data and Model Heterogeneity-Aware Federated Learning via Dynamic Weight Sharing

Federated Learning (FL) plays an indispensable role in edge computing systems. Prevalent FL methods mainly address challenges involved in heterogeneous data distribution across devices. Model heterogeneity, however, has seldom been put under scrutiny. In practice, different devices (e.g., PCs and smartphones) generally have disparate computation and communication resources, necessitating neural network models with varying parameter sizes. Therefore, we propose FedDM, a novel data and model heterogeneity-aware FL system, which improves the FL system's accuracy while reducing edge devices' computation and communication costs for heterogeneous model training. FedDM features: 1) dynamic weight sharing scheme that handles model heterogeneity by dynamically selecting parts of the large model to share with smaller ones; 2) tree-structured layer-wise client cooperation scheme that handles data heterogeneity by allowing clients with similar data distribution to share some network layers. We implement FedDM and evaluate it using five public datasets with different tasks.