LCwmcaR: Learning Cross-Window Cross-Modality Correlation-Aware Representation for Human Activity Recognition

Deep learning (DL)-based human activity recognition (HAR) has attracted considerable attention owing to its vast potential across various applications. Currently, HAR still faces two challenges. For one thing, existing methods neglect the spatial distribution information embedded in HAR signals and lack the ability to comprehensively model the spatial–temporal (ST) dependencies within HAR data, restricting them from effectively decoding human activity. For another thing, previous models generate feature representations for a sliding window of the sequence solely based on this window itself, without cross-window interaction learning, posing challenges to classifiers, such as perceptual aliasing or feature inconsistency issues. For that, we propose a novel cross-window and cross-modality correlation-aware framework, namely LCwmcaR, which is a dual-branch network that simultaneously models temporal- and spatial-level information using Mamba and convolutional neural network (CNN), respectively. Additionally, a learnable temporal 2-dimensionalization (LT2D) strategy is designed to encode low-level temporal patterns into high-level learnable image-liked 2-D space representations that integrate both local and global ST dependencies. Moreover, a cross-window correlation-aware feature representation generation (CrwcaFRGen) module, which correlates multiple windows representations within a batch at the attention level, is introduced to produce more robust features for the classifier. Experimental results on four public datasets demonstrate that the LCwmcaR outperforms state-of-the-art (SOTA) methods by a large margin.