RESTORE-DiT: Reliable satellite image time series reconstruction by multimodal sequential diffusion transformer

Repetitive optical observations from satellites are crucial for monitoring earth surface dynamics over time. However, optical satellite image time series is severely affected by frequent data gaps due to clouds and shadows. While synthetic aperture radar (SAR) provides cloud-penetrating capabilities to complement missing optical data, recent advancements in time series reconstruction have shifted focus from incorporating single SAR image to exploiting SAR time series. However, current methods still struggle for challenging scenarios like highly dynamic surface, persistent data gaps, and exhibit poor resilience to inaccurate cloud masks. In this research, we approach the time series reconstruction problem from the perspective of conditional generation. We propose a multimodal diffusion framework termed RESTORE-DiT, which firstly promotes the sequence-level optical-SAR fusion through a diffusion framework. Specifically, date-matched SAR time series provide under-cloud surface dynamics to guide the denoising process of cloudy areas, and date information is embedded to account for irregular observation intervals and periodic patterns. Extensive experiments on three regions have shown the proposed method achieves state-of-the-art performance. RESTORE-DiT outperforms comparison methods by 2.87 dB in PSNR and a 27.2 % reduction in RMSE on France site. SAR and date information together increase PSNR by 2.41 dB. The reconstructed optical image time series is verified to accurately reflect the crop growth condition and support for long-term vegetation observations. In addition, RESTORE-DiT can be easily extended to other conditional reconstruction or prediction tasks for arbitrary time series image data, thus facilitating spatiotemporal analysis research. The codes will be public available at: https://github.com/SQD1/RESTORE-DiT.