Building safe, effective, and enjoyable autonomous vehicles is a lot harder than has historically been considered. Fully autonomous driving systems need to be able to handle the long tail of real world situations, which requires sensors, algorithms, and data collection being sufficiently developed to “solve” the full driving task. The goal of our research is to develop, through large-scale real-world driving data, vision-based AI systems that enhance driving safety and lead to fully autonomous driving. In this talk, I will present our work on the MIT DriveSeg dataset including 1) collect and annotate large-scale driving scene segmentation datasets 2) study the value of temporal dynamics in the segmentation task, and 3) explore unsupervised clustering and edge-case discovery methods. I will also briefly talk about the MIT Advanced Vehicle Technology (MIT-AVT) Study for large-scale naturalistic driving data collection and analysis.