Image- and sequence- data are essential for autonomous driving. A release of AD products is unimaginable without excessive training and validation data for perception stacks. Quantity and variance have been the main KPI as it has been assumed that quantity helps to overcome the “coverage problem” for an incredible variant world. Accidents with self driving cars reveal that engineers did not yet cover the potential parameter variance required to react correctly in all situations. We report about an approach to find systematic data gaps which exist in spite of long “situation checklists”, “weather matrix”, “object catalogues”. Based on a “world variation matrix” which covers the properties of relevant domains from object -shape -dimensions -poses -positions -surface characteristics -materials -behavior illuminants – spectral composition, -temporal behavior, -intensity, spatial distribution detector properties -spectral-,spatial-, temporal- resolution detection probabilities and all their permutations, we estimated the complexity for perception task in this context. The efforts to sample this huge parameter space by driving around recording the “real world”, add simulated situation and generate virtual data still leave gaps in the data set which might limit a “fail safe operation”.
We share an approach to learn from fails by extracting mutual effect chains and systematically verify and improve the data base by manipulation or introduction of additional synthetic contents into the data sets. Doing this we always have an eye on the source of deficiencies and try to identify “detection gaps” potentially caused by deficiencies of the used sensor set.