TADEL: Task-Aware Dynamic Ensemble of Lightweight LLMs for Improved Inference Accuracy in Edge AI

The proliferation of large language models (LLMs) has significantly advanced the capabilities of automated reasoning and language understanding across a wide range of domains. However, their deployment in edge AI environments remains constrained by limited computational resources and strict latency requirements. Although prior techniques have focused on accelerating inference through model compression or quantization, these methods frequently result in reduced predictive accuracy, particularly when handling tasks that span diverse subject areas. This work introduces TADEL, an adaptive LLM ensemble framework that addresses this gap by combining fast, lightweight models with a dynamic weighting mechanism that enhances accuracy under resource constraints. By casting the problem of model ensembling as a Markov Decision Process, we employ a reinforcement learning (RL) approach based on Proximal Policy Optimization (PPO) to regulate the contribution of each model during ensemble decision making. The learning policy incorporates semantic task attributes, enabling it to generate context-sensitive weightings that respond effectively to varying workloads. Extensive evaluations are conducted on MMLU, CMMLU, and GSM8K benchmarks covering a broad spectrum of academic and professional domains. The results indicate that TADEL achieves an improvement of up to 11.2% in overall accuracy compared to naive weighted-ensemble baseline and exceeds the performance of the standalone model by more than 11.5%.