The superior performance of CNN on medical image analysis heavily depends on the annotation quality, such as the number of labeled images, the source of images, and the expert experience. The annotation requires great expertise and labor. To deal with the high inter-rater variability, the study of the imperfect label has great significance in medical image segmentation tasks. In this paper, we present a novel cascaded robust learning framework for chest X-ray segmentation with imperfect annotation at the boundary. Our model consists of three independent networks, which can effectively learn useful information from peer networks. The framework includes two stages. In the first stage, we select the clean annotated samples via a model committee setting, the networks are trained by minimizing a segmentation loss using the selected clean samples. In the second stage, we design a joint optimization framework with label correction to gradually correct the wrong annotation and improve the network performance. We conduct experiments on the public chest X-ray image datasets collected by Shenzhen Hospital. The results show that our methods could achieve a significant improvement on the accuracy in segmentation tasks compared to the previous methods.