Spatio-Temporal Neural Rendering: Reversible Time-to-Vision Representation Learning for Human Activity Recognition

Recently, wearable human activity recognition (WHAR) has attracted considerable attention, driven by its diverse applications in healthcare monitoring, smart environments, and human-computer interaction. Despite advances, a key challenge still limits the further development of existing WHAR methods: Individual time points contain less semantic information, and it is challenging for single-domain representation learning to reliably discern spatial-temporal (ST) dependencies hidden by inherently intricate hierarchical temporal patterns of WHAR data. To this end, we introduce a novel perspective and design innovation, Spatio-Temporal Neural Rendering (STNR), which is a vision-centric dual-domain framework tailored for WHAR that establishes reversible time-to-vision transformations between temporal data and learnable visual-like representations. Specifically, a Temporal-Visual Duality Mapping (TVDM) module encompassing a 2D rendering pathway and a 1D inverse rendering pathway is designed to unify temporal and visual modalities by leveraging their complementary strengths for enhanced WHAR. In addition, a dual-pathway Dynamic Mixing Layer (DML) is introduced to exploit both temporal variations and cross spatio-temporal patterns embedded within WHAR data. Extensive experiments conducted across six widely used WHAR datasets demonstrate that our proposed STNR achieves state-of-the-art (SOTA) performance and can be generalized to gait recognition task, providing a brand new paradigm for cross-domain time series research.