VTGNet: A Vision-based Trajectory Generation Network for Autonomous Vehicles in Urban Environments

Peide Cai, Yuxiang Sun, Hengli Wang, Ming Liu

Research output: Journal article publicationJournal articleAcademic researchpeer-review

14 Citations (Scopus)


Traditional methods for autonomous driving are implemented with many building blocks from perception, planning and control, making them difficult to generalize to varied scenarios due to complex assumptions and interdependencies. Recently, the end-to-end driving method has emerged, which performs well and generalizes to new environments by directly learning from export-provided data. However, many prior methods on this topic neglect to check the confidence of the driving actions and the ability to recover from driving mistakes. In this paper, we develop an uncertainty-aware end-to-end trajectory generation method based on imitation learning. It can extract spatiotemporal features from the front-view camera images for scene understanding, then generate collision-free trajectories several seconds into the future. The experimental results suggest that under various weather and lighting conditions, our network can reliably generate trajectories in different urban environments, such as turning at intersections and slowing down for collision avoidance. Furthermore, closed-loop driving tests suggest that the proposed method achieves better cross-scene/platform driving results than the state-of-the-art (SOTA) end-to-end control method, where our model can recover from off-center and off-orientation errors and capture 80% of dangerous cases with high uncertainty estimations.

Original languageEnglish
JournalIEEE Transactions on Intelligent Vehicles
Publication statusAccepted/In press - 2020


  • Autonomous vehicles
  • Cameras
  • End-to-end driving model
  • Feature extraction
  • imitation learning
  • path planning
  • Planning
  • Task analysis
  • Trajectory
  • Uncertainty
  • uncertainty-aware visual navigation

ASJC Scopus subject areas

  • Artificial Intelligence
  • Automotive Engineering
  • Control and Optimization


Dive into the research topics of 'VTGNet: A Vision-based Trajectory Generation Network for Autonomous Vehicles in Urban Environments'. Together they form a unique fingerprint.

Cite this