Ultrasound volume projection imaging (VPI) has shown to be appealing from a clinical perspective, because of its harmlessness, flexibility, and efficiency in scoliosis assessment. However, the limitations in hardware devices degrade the resultant image content with strong structured noise. Owing to the unavailability of reference data and the unpredictable degradation model, VPI image recovery is a challenging problem. In this paper, we propose a novel framework to learn the structured noise removal from unpaired samples. We introduce the attention mechanism into the generative adversarial network to enhance the learning by focusing on the salient corrupted patterns. We also present a dual adversarial learning strategy and integrate the denoiser with a segmentation model to produce the task-oriented noiseless estimation. Experimental results show that the proposed method can improve both the visual quality and the segmentation accuracy on spine images.