SMART: Stratified Matching and Recurrent Transformer for Optical Flow Estimation

The current optical flow estimation method GMFlow, which combines a hierarchical refinement strategy with iterative refinement, has achieved very good performance. However, it struggles to handle frames in complex scenes well, mainly because of unreliable coarse predictions. In this paper, we present a Transformer-based parallel refinement network to improve the accuracy of coarse predictions and allow fine predictions to be adjusted, based on the accurate coarse positional information. Our proposed structure, called SMART, maximizes the utilization of coarse-level rich-information features that are discarded after global matching in GMFlow. Additionally, the parallel structure allows the coarse-level prediction to be refined throughout the process and updated with information from both levels. Experimental results show that our model outperforms the baseline on two important datasets, namely FlyingChairs and FlyingThings3D.