Because of long distance of railway lines, it is difficult to find an appropriate method to inspect the rail condition efficiently and accurately. In this paper, a new technique based on driving recorder and image analysis is proposed to evaluate the rail condition. In line with the videos of driving recorder, three coordinate systems for train, rail, and 'observer' are defined in the Lagrangian space, respectively. The relationship between the three coordinate systems is then figured out to facilitate the identification of rail condition such as rail curvature, rail irregularity, and rail vehicle swing, etc. Based on the identified rail condition status by the proposed technique, the assessment of derailment, damage at rail fasteners, and contact condition between the wheel and rail can be conducted. In the case study, a set of videos from the driving recorders of trains during their in-service operations are analyzed by the proposed technique. The results show that the proposed technique is competent to evaluate the rail curvature, rail irregularity, vehicle swing, and train running speed accurately. Moreover, the location where the contact between the wheel and rail is abnormal can be identified by the proposed technique. It is testified that the proposed technique provides a simple and accurate way to inspect the rail condition.