Comprehensive understanding of dynamic environments is a critical prerequisite for autonomous vehicles. In this talk, I will discuss state-of-the-art techniques for panoptic segmentation which enables self-driving cars to simultaneously recognize instances of traffic participants as well as scene semantics of the background in an efficient manner. I will then introduce a new perception task that denoted as Multi-Object Panoptic Tacking (MOPT) which allows for modelling dynamic scenes holistically by jointly learning panoptic segmentation and multi-object tracking in an end-to-end network.