To facilitate condition assessment in sewer pipeline networks current practice is using the available technologies to visually inspect the internal condition of pipelines. Closed circuit television (CCTV) has been one of the most used methods in North American municipalities in last decades. However, this method requires hours of videos to be inspected by certified inspectors which is time consuming, labor intensive, and error prone. The main objective of this research is to propose an automated approach for inspection and condition assessment of sewer pipelines using computer vision techniques. This research includes two main part: Identifying region of interest (ROI) in sewer inspection videos which are most likely to contain sewer defects, and defect detection and classification among the identified anomalous frames. The ROI detection model employs proportional data modeling using hidden Markov models (HMM) to extract abnormal frames from sewer CCTV videos. In the next step, a deep learning approach using convolutional neural networks (CNN) is proposed to detect the defects and classify them. The presented algorithm has been developed and tested using the data sets from CCTV inspection reports.